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The  review  of  the  current  status  of  the  Na¬ 
tional  Science  Foundation's  management 
information  system  and  its  plans  for  making 
needed  improvements  focuses  on  identifying 
changes  planned  or  in  process,  quality  con¬ 
trols,  user  satisfaction,  and  cost. 

Reliability  problems,  including  inaccurate  and 
incomplete  data  and  slow  response  time  at 
computer  terminals,  seriously  impair  the 
system's  effectiveness  and  frustrate  its  users. 
Recommendations  for  management  and  tech¬ 
nical  improvements  made  by  a  Foundation 
consultant  3  years  ago  have  not  been  carried 
out.  A  September  1979  study  of  system  per¬ 
formance  also  pointed  out  the  need  for  many 
system  improvements. 

The  Foundation  can  increase  system  reliabil¬ 
ity  by  improving  quality  control,  system  re¬ 
sponse  time,  long-range  planning,  perform¬ 
ance  evaluation,  and  administrative  manage¬ 
ment.  The  Foundation  also  should  determine 
users'  needs,  and  establish  specific,  quantified 
goals  and  target  dates  for  correcting  defi- 
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COMPTROLLER  GENERAL  OF  THE  UNITED  STATES 
WASHINGTON.  D.C.  tOMt 


The  Honorable  Don  Fuqua 
Chairman,  Committee  on  Science 
and  Technology 
House  of  Representatives 

Dear  Mr.  Chairman: 

In  accordance  with  your  Committee's  request  and 
subsequent  discussions  with  our  representatives,  we  reviewed 
the  current  status  of  the  National  Science  Foundation's  man¬ 
agement  information  system  (MIS)  and  the  Foundation's  plans 
for  making  needed  improvements.  The  review  focused  on  identi¬ 
fying  major  changes  planned  or  in  process,  quality  controls, 
user  satisfaction,  and  cost.  As  agreed,  we  did  not  conduct 
a  major  audit  of  MIS  data  quality  because  we  found  early 
indications  of  system  and  management  problems  that  would 
likely  result  in  unreliable  data.  Nor  did  we  include  in  our 
audit  scope  a  determination  of  how  well  actual  MIS  operations 
conformed  tc  stated  procedures.  Such  audit  work  would  have 
been  unproductive,  in  our  view,  because  system  documentation 
and  procedures  were  not  always  available  or  current. 

We  reviewed  a  1977  management  consultant's  study  of  MIS 
to  determine  how  the  system  functioned  then,  and  what  im¬ 
provements  had  been  recommended.  We  then  determined  how 
the  Foundation  had  acted  on  these  recommendations.  We  used 
a  General  Accounting  Office  standardized  format  to  assess 
reliability,  determined  user  satisfaction  from  user  question¬ 
naires  based  on  several  MIS  routine  reports,  interviewed  a 
sampling  of  Foundation  personnel  who  manage  the  system  or 
use  it  regularly,  and  made  test  checks  of  data  from  several 
subsystems  by  comparing  automated  data  with  original  docu¬ 
ments.  We  also  gathered  system  cost  data  from  Foundation 
personnel.  Finally,  we  reviewed  a  report  on  a  system  per¬ 
formance  study  conducted  during  1979  by  the  Federal  Computer 
Performance  Evaluation  and  Simulation  Center.  At  the  request 
of  ycur  office,  we  delayed  issuance  of  this  report  until  we 
could  evaluate  the  Center's  study  which  was  published  after 
most  of  our  work  was  done. 

We  obtained  Foundation  officials'  views  on  our  findings 
and  recommendations,  and  their  comments  are  considered  in 
the  report.  Our  findings  are  summarized  below  and  discussed 
in  more  detail  in  the  appendix. 
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SYSTEM  DESCRIPTION  AND  MANAGEMENT 

The  goal  of  MIS  is  to  automate  all  the  Foundation's 
administrative  and  program  functions.  MIS  is  made  up  of  5  . 
major  systems  which  are  divided  into  20  subsystems  with  specif¬ 
ic  Foundation  functions.  All  the  subsystems  use  data  from  a 
common  data  base  (integrated  data  base)  so  that  data  must  be 
entered  only  once  into  the  system.  A  Honeywell  computer  system 
provides  access  to  the  data  from  284  terminals  throughout  the 
Foundation's  headquarters  and*  annexes.  Users  of  the  system's 
data  include  Foundation  management,  the  Congress,  the  Office 
of  Management  and  Budget,  and  grantee  institutions. 

A  staff  of  128  Foundation  and  contractor  personnel 
administer ‘  the  system,  which  is  managed  by  the  Directorate  for 
Administration  through  the  Division  of  Information  Systems 
(DIS).  The  Foundation  began  developing  MIS  in  1971  and 
has  spent  over  $15  million  on  it  through  fiscal  year  1979, 
excluding  personnel  costs. 

MANAGEMENT  CONSULTANT'S  EVALUATION 

A  management  consultant  assessed  the  system  in  1976  and 
found  that  user  objectives  had  not  been  met  and  that  signif¬ 
icant  improvements  in  the  system  were  needed.  The  consul¬ 
tant  recommended  changes  and  provided  an  action  plan  for 
future  work.  Since  the  consultant’s  study,  systems  and  man¬ 
agement  changes  made  to  improve  the  system  include  establish¬ 
ment  of  a  steering  committee,  user  representatives,  director¬ 
ate  task  groups,  and  a  management  controls  branch;  develop¬ 
ment  of  a  financial  accounting  system  implementation  plan; 
and  redesign  and  enhancements  to  several  subsystems.  The 
Foundation  has  .not  studied  the  impact  of  these  changes,  but 
system  managers  and  users  find  these  changes  useful.  How¬ 
ever,  some  of  the  consultant's  recommendations  for  changes 
in  important  areas  such  as  data  quality  assessment,  long- 
range  planning,  and  system  performance  evaluation  have  not 
been  implemented. 

THE  SYSTEM  AND  ITS  MANAGEMENT 
STILL  NEED  IMPROVEMENT 


While  MIS  is  not  yet  fully  implemented,  it  is  being  used 
to  supply  data  important  to  managing  the  Foundation.  However, 
users  still  cite  system  reliability  problems,  particularly 
inaccurate  and  incomplete  data,  and  slow  response  time  at  the 
computer  terminals.  To  solve  these  and  other  problems,  improve¬ 
ments  are  needed  in  five  areas — quality  control,  system  re¬ 
sponse  time,  long-range  planning,  performance  evaluation,  and 
administrative  management.  Each  of  these  areas  is  discussed 
briefly  below. 
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The  Foundation  does  not  have  procedures  to  monitor  data 
quality  or  identify  system  problems  which  cause  poor  data 
quality.  The  consultant  recommended  in  his  January  1977  report 
that  the  Foundation  immediately  conduct  a  data  quality  study 
to  identify  system  weaknesses  and  develop  data  quality  control 
procedures.  However,  the  Foundation  deferred  the  study  be¬ 
cause  of  higher  priority  work  and  because  system  managers  be¬ 
lieved  the  data  was  accurate  despite  the  absence  of  formal 
quality  control  procedures. 

Our  reliability  assessment  revealed  some  system  weaknesses 
that  reduce  reliability.  The  weaknesses  include  inadequate 
error  correction  procedures  to  insure  the  timely  and  accurate 
correction  of  data  errors  that  are  found  in  the  data  base, 
system  design  problems  which  make  identifying  and  correcting 
data  errors  difficult,  and  inadequate  system  documentation 
which  make  identifying  and  correcting  system  design  problems 
difficult. 

System  users  complain  that  response  time  at  MIS  computer 
terminals  is  frequently  slow.  Reported  delays  of  several 
minutes  when  numerous  terminals  are  concurrently  active  indi¬ 
cate  a  probable  system  overload.  Foundation  officials  are 
aware  that  this  problem  has  existed  for  at  least  2  years. 

In  February  1979,  the  Federal  Computer  Performance  Eval¬ 
uation  and  Simulation  Center  (FEDSIM)  began  assessing  the 
design  and  performance  of  the  Foundation's  computer  system 
including  system  response  time.  The  study  showed  that  oc¬ 
currences  of  slow  system  response  time  were  not  primarily 
traceable  to  system  hardware,  but  rather  to  the  management 
of  available  computer  resources.  FEDSIM  stated  that  improv¬ 
ing  operating  procedures,  workload  scheduling,  and  software 
design  could  minimize  the  impact  of  heavy  prime  time  user 
demand  for  computer  resources  and  improve  system  response 
time.  Also,  the  Foundation  needs  better  system  accounta¬ 
bility  data  to  improve  its  use  of  available  computer  re¬ 
sources.  Meanwhile,  the  Foundation  added  46  more  terminals 
to  the  system  in  fiscal  year  1979. 

The  Foundation's  MIS  management  program  lacks  both  a 
long-range  plan  and  a  planning  process.  The  consultant  cited 
the  absence  of  effective  long-range  planning  as  adversely 
affecting  system  management  and  recommended  that  the  Founda¬ 
tion  pursue  immediately  a  long-range  plan  and  a  planning,  pro¬ 
cess.  The  Foundation  still  has  no  firm  plan  for  developing 
a  long-range  system  plan,  and  consequently  has  no  firm  target 
date  or  projected  cost  for  completing  the  system. 

The  system  is  now  managed  on  a  short-range  basis,  es¬ 
sentially  reacting  to  daily  problems.  While  numerous  work 
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projects  for  system  improvements  have  been  identified,  there 
were  often  no  detailed  documented  plans  for  the  improve¬ 
ments  nor  funds  to  put  them  into  effect. 

The  Foundation  has  no  procedures  for  systematically 
assessing  user  needs  and  evaluating  system  performance  and 
cost  in  meeting  those  need^.  Since  the  MIS  initial  require¬ 
ments  study  in  1971,  the  Division  of  Information  Systems 
has  assumed  the  task  of  determining  the  current  needs  of 
program  users.  In  so  doing,  system  managers  have  used  feed¬ 
back  from  program  personnel  through  user  representatives  and 
task  groups.  The  consultant's  recommendations  for  performance 
evaluation  and  cost  accounting  programs  have  not  been  imple¬ 
mented  . 

We  found  that  other  aspects  of  overall  system  management 
need  improving,  such  as  system  security,  documentation,  and 
training  for  users  and  managers.  Also,  the  Foundation's 
internal  audit  staff,  which  should  monitor  the  MIS  operation, 
lacks  the  technical  capability  to  evaluate  the  system  ef¬ 
fectively. 

CONCLUSIONS 


Significant  improvements  in  MIS  and  its  management  are 
needed  if  the  system  is  to  fulfill  its  intended  purpose  and 
achieve  a  degree  of  reliability  satisfactory  to  its  users. 
Although  3  years  have  passed  since  the  consultant's  evaluation 
study,  the  Foundation  has  not  yet  implemented  major  recom¬ 
mendations  that  the  consultant  said  were  needed  immediately. 

As  a  result,  serious  technical  and  management  problems 
threaten  the  successful  operation  of  the  system.  System 
managers  cite  lack  of  resources  as  the  main  reason  for  not 
implementing  important  recommendations.  However,  we  believe 
that  the  absence  of  long-range  planning  and  a  knowledge  of 
system  effectiveness  has  caused  the  Foundation  to  give  low 
priority  to  areas  such  as  data  quality  and  system  reliability 
which  should  have  had  the  highest  priority. 

The  Foundation  is  adding  additional  equipment  to  the 
system,  which  is  already  experiencing  response  time  delays, 
without  knowing  what  adverse  impacts  this  equipment  will 
have  on  the  system's  overall  performance.  Considering  the 
absence  of  good  system  accountability  data  and  FEDSIM's 
conclusion  that  system  hardware  is  not  a  constraint  to  im¬ 
proved  system  performance,  we  believe  the  Foundation  should 
carefully  assess  any  further  plans  for  adding  terminals  and 
associated  equipment  to  the  system.  Better  management  of 
existing  computer  resources,  as  recommended  by  the  consultant 
and  FEDSIM,  would  seem  to  be  a  more  desirable  alternative  at 
this  time. 
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The  value  of  'MIS  to  the  Foundation,  and  user  ac¬ 
ceptance  of  it,  depends  on  high  system  reliability.  Our 
reliability  assessment  shows  the  system  to  be  of  medium 
reliability,  which  means  that  users  of  system  data  should 
verify  the  accuracy  of  the  data  before  using  it  for  impor¬ 
tant  decisions.  In  our  view,  system  users  within  the 
Foundation,  the  Congress,  the  Office  of  Management  and  Budget 
and  others  will  not,  nor  should  they,  be  satisfied  with  data 
from  a  medium-reliability  system. 

While  MIS  is  not  yet  fully  implemented,  it  is  already 
being  depended  upon  to  provide  many  essential  services 
and  data  to  Foundation  management.  Yet  the  system  will  be 
unable  to  provide  highly  reliable  data  until  the  technical 
and  management  problems  that  we  identified  in  this  report 
are  addressed  and  remedied.  In  addressing  these  problems, 
we  believe  the  Foundation  should  re-examine  the  purpose 
of  MIS,  clearly  define  its  users  and  their  needs,  and 
establish  specific,  quantified  goals  and  target  dates  for 
correcting  deficiencies  and  completing  the  system. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 


We  proposed  that  the  Foundation  take  several  actions 
to  improve  MIS  including  actions  to  resolve  the  slow  sys¬ 
tem  response  time  problem,  provide  data  quality  control, 
and  assess  user  needs.  In  a  letter  dated  June  15,  1979, 
the  Foundation  advised  that  although  it  had  concerns  about 
our  reliance  on  the  use  of  questionnaires  and  interviews 
to  assess  user  satisfaction  with  MIS,  it  generally  agreed 
with  most  of  our  proposed  recommendations.  However,  the 
Foundation  did  not  agree  that  it  had  a  slow  system  response 
time  problem  of  the  magnitude  we  described  in  the  draft  re¬ 
port.  The  Foundation  referred  to  FEDSIM's  study  of  MIS's 
user  response  time  which,  the  Foundation  said,  showed 
response  time  to  be  in  an  acceptable  range. 

The  Foundation  suggested  that  our  user  satisfaction 
findings  regarding  two  widely  used  reports  produced  by  MIS 
could  not  be  attributed  solely  to  the  quality  of  MIS's 
integrated  data  base,  but  rather  may  have  been  the  result 
of  differences  in  user  perception  of  the  reports.  We  be¬ 
lieve  our  questionnaires  and  interviews  identified  a  legi¬ 
timate  system  problem — errors  in  the  data  base — which  ap¬ 
pears  to  adversely  affect  the  accuracy  of  a  widely  used  re¬ 
port  produced  from  the  data  base.  We  selected  two  widely 
used  reports  produced  by  MIS,  each  of  which  is  used  by  a 
different  Foundation  management  group.  Program  managers  use 
the  Workload  Status  Report,  whereas  division  directors  use 
the  Proposal  Aging  Report.  We  asked  only  program  managers 
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about  the  Workload  Status  Report,  while  only  division  director 
were  asked  about  the  Proposal  Aging  Report.  Our  questionnaire 
showed  that  users  of  the  Proposal  Aging  Report  (division 
directors)  were  having  problems  with  the  report  because  of 
errors,  and  the  Foundation  confirmed  this  by  noting  that  of 
the  two  reports  we  tested,  more  errors  occur  in  the  Proposal 
Aging  Report. 

The  Foundation  suggested'  in  its  letter  that  although 
errors  are  present  in  the  Workload  Status  Report,  the  errors 
are  not  as  easily  recognized  as  in  the  Proposal  Aging  Report 
because  of  the  larger  volume  of  information  present  in  the 
workload  report.  We  believe  errors  in  the  data  base  are 
still  errors,  not  just  perceptions,  no  matter  how  unrecog¬ 
nizable  they  might  be  in  some  reports.  The  Foundation  also 
suggested  that  the  use  of  the  Proposal  Aging  Report  by  manage¬ 
ment  as  an  oversight  tool  over  program  managers  may  have 
caused  the  managers  to  resent  the  aging  report,  and  judge 
the  report  more  harshly  in  our  user  survey.  However,  we 
did  not  obtain  program  managers'  views  on  their  satisfaction 
with  the  Proposal  Aging  Report — program  managers  were  not 
asked  to  judge  this  report.  We  asked  only  division  directors 
— who  are  the  primary  users  of  the  aging  report — how  satis¬ 
fied  they  were  with  it. 

The  Foundation  stated  in  its  letter  that  the  FEDSIM 
tests  show  system  response  time  to  be  in  an  acceptable 
range  and  that  the  tests,  as  well  as  the  agency's  recent 
experience,  do  not  support  the  implication  in  the  GAO  re¬ 
port  that  only  20  terminals  can  be  concurrently  active  on 
the  system.  The  Foundation's  letter  was  dated  June  15,  1979, 
prior  to  FEDSIM's  final  report.  The  final  report,  dated 
September,  1979',  acknowledges  the  occurrence  of  slow  system 
response  times  and  does  not  characterize  current  system 
performance  as  being  either  acceptable  or  unacceptable. 
FEDSIM's  report  shows  that  hardware  performance  was  generally 
good  during  the  controlled  testing  and  was  not  a  constraint 
to  improved  system  performance,  but  that  better  management 
of  available  computer  resources  would  improve  system  perform¬ 
ance. 


We  conclude  from  the  FEDSIM  report  that  there  are  times 
when  system  response  time  is  poor.  The  study  indicates  that 
under  controlled  testing  it  is  possible  to  have  more  than 
20  terminals  concurrently  active.  However,  the  day  to  day 
operations  of  the  Foundation  are  a  real  world  environment 
where  the  computer  system  must  respond  to  uncontrolled  de¬ 
mands  which  we  believe  can  yield  situations  where  only  20 
terminals  could  be  concurrently  active,  or  slow  system  re¬ 
sponse  times  could  occur. 
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Regardless  of  the  cause,  users  will  react  negatively  to 
slow  response  time  at  the  terminals  even  if  they  are  apprised 
of  possible  causes.  The  Foundation’s  task,  as  we  see  it,  is 
to  eliminate  or  minimize  what  Foundation  personnel  perceive' 
as  a  system  response  time  problem  regardless  of  the  exact 
number  of  terminals  involved.  We  believe  FEDSIM's  recommenda¬ 
tions  should  be  helpful  in  doing  so. 

Although  the  Foundation  states  in  its  letter  that  it  has 
taken  steps  to  address  user  needs  by  establishing  an  advisory 
group,  we  believe  the  Foundation  should  insure  that  the 
group's  actions  will  be  adequate  to  provide  for  user  needs 
assessment.  As  we  have  recommended,  a  performance  evaluation 
program  is. also  needed  in  addressing  user  needs. 

RECOMMENDATIONS 

We  recommend  that  the  Director  of  the  National  Science 
Foundation  improve  MIS  by  taking  actions  in  the  following 
areas: 

— System  Response  Time — Resolve  expeditiously  the  slow 
system  response  time  problem  which  is  frustrating 
system  users,  and  reassess  the  need  for  acquiring 
additional  hardware. 

— Data  Quality  Control — Assess  MIS's  data  quality 
and  establish  procedures  for  periodically  testing 
and  maintaining  data  quality  to  insure  high  re¬ 
liability. 

— Long-Range  Planning — Develop  and  maintain  a  long- 
range  system  and  hardware  planning  process  to  insure 
orderly  and  systematic  development  of  MIS.  The  plan¬ 
ning  process  should  include  top  management  partici¬ 
pation;  provide  a  reasonable  5-year  projection  of 
system  requirements  and  costs,  which  would  ihrcludV*  ......... 

life-cycle  cost  analysis;  establish  quantified  goal^- 
and  priorities  for  work  projects;  and  set  ’milestone- 
completion  dates. 

— Performance  Evaluation — Develop  and  implement  perform- - 
ance  evaluation  procedures  to  determine  and  control 
the  efficiency  of  the  system  in  meeting  its  goals.' 

Such  procedures  should  assess  user  needs  and  insure 
system  cost  effectiveness. 

— Overall  System  Management — Assess  MIS  short-  and  mid¬ 
range  management  efforts  to  insure  that  areas  such  as 
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general  system  security/  system  documentation/  and 
training  receive  adequate  attention.  Also/  the 
Director  should  strengthen  the  internal  audit  staff 
capability  to  monitor  MIS's  performance. 

As  agreed  with  your  representatives,  our  office  will  limit 
initial  distribution  of  this  report  to  House  Science  and  Tech¬ 
nology  Committee  and  National  Science  Foundation  recipients. 
Subsequent  distribution  will  be  made  to  interested  parties  1 
week  from  the  date  of  the  report. 


Sincu;.: v^ly  yours 

Comptroller  General 
of  the  United  States 
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THE  NATIONAL  SCIENCE  FOUNDATION'S 
MANAGEMENT  INFORMATION  SYSTEM:  A  STATUS  REPORT 

INTRODUCTION 


On  July  20,  1978,  the  Chairman  of  the  House  Committee 
on  Science  and  Technology  requested  that  we  review  the  Nation¬ 
al  Science  Foundation's  management  information  system  (MIS). 
The  Chairman  was  concerned  with  the  system's  data  quality  and 
reports  from  Foundation  staff  that  the  system  had  not  been 
entirely  satisfactory.  He  stated  that  an  assessment  of  the 
current  status  of  MIS  and  the  Foundation's  plans  for  making 
needed  improvements  would  be  useful  to  the  Committee  in  carry¬ 
ing  out  its  oversight  of  the  Foundation.  The  Chairman  asked 
that  the  assessment  consider: 

— identifying  major  changes  planned  and  in  process; 

— identifying  and  testing  quality  controls  for  assuring 
data  accuracy; 

— determining  the  Foundation's  efforts  to  identify  the 
needs  of  the  system's  users  and  its  plans  to  meet 
those  needs;  and 

— identifying  the  cost  of  MIS  and  of  any  planned 
major  purchases  of  hardware  or  software. 

Pursuant  to  the  Chairman's  request  and  later  agreements 
with  our  representatives,  we: 

— reviewed  a  1977  consultant's  study  of  MIS  to  under¬ 
stand  how  the  system  functioned  then,  what  problems 
were  identified  and  recommendations  made;  and  what 
actions  the  Foundation  had  taken  on  these  recommenda- 
ations; 

— interviewed  a  cross  section  of  Foundation  personnel 
who  manage  the  system  and  use  it  regularly  to  determine 
how  the  system  functions  today  and  how  it  is  managed; 

— performed  a  system  reliability  assessment  using  our 
"Audit  Guide  for  Reliability  Assessment  of  Controls 
In  Computerized  Systems"; 

— made  limited  test  checks  of  data  accuracy  in  three  key 
subsystems  by  comparing  automated  data  with  original 
documents; 
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— determined  user  satisfaction  from  questionnaires  based 
on  several  MIS  routine  reports; 

— gathered  system  cost  data;  and 

— reviewed  a  report  on  a  system  performance  study  con¬ 
ducted  during  1979  by  the  Federal  Computer  Performance 
Evaluation  and  Simulation  Center  (FEDSIM). 

In  assessing  the  status  of  MIS  we  limited  our  audit  work 
to  assessing  the  system  through  interviews,  questionnaires, 
and  spot  testing  of  data.  As  agreed  with  the  House  Committee, 
we  did  not  attempt  the  time-consuming  effort  to  conduct  a  major 
audit  of  MIS  data  quality  because  we  found  early  indications 
of  system  and  management  problems  that  would  likely  result  in 
unreliable  data.  Nor-  did  we  include  in  our  audit  scope  a 
determination  of  how  well  actual  MIS  operations  conformed  to 
stated  procedures.  Such  audit  work  would  have  been  unpro¬ 
ductive,  in  our  view,  because  system  documentation  and  pro¬ 
cedures  were  not  always  available  or  current. 

System  development 

The  Foundation,  under  the  authority  of  the  National 
Science  Foundation  Act  of  1950,  as  amended  (42  U.S.C.  1861  et 
seq. )  supports  scientific  research  in  various  disciplines 
(such  as  chemistry,  physics,  biology,  and  engineering)  pri¬ 
marily  by  awarding  grants  to  colleges  and  universities.  The 
Foundation  processes  about  28,000  research  proposals  annually. 
Of  these,  approximately  24,000  are  acted  on  and  eventually 
about  $900  million  is  paid  out  in  grants. 

To  help  process  proposals  and  to  provide  current  data 
for  program  management,  the  Foundation  decided  in  1970  to 
automate  all  administrative  and  program  functions  by  using 
an  MIS.  The  project  began  in  1971  with  a  major  requirements 
study  to  identify  all  the  functions  the  system  would  perform. 
This  study  was  followed  by  a  system  design  phase  and  the 
installation  of  the  system.  By  1975,  all  major  subsystems 
had  been  installed  although  some  were  running  parallel  with 
older  manual  systems  and  some  subsystems,  such  as  the  financial 
accounting  subsystem,  had  serious  operational  problems.  In 
1977,  an  office  automation  project  extended  the  system  to 
include  automated  word  processing. 

While  the  Foundation  had  automated  parts  of  its  program 
before  1971,  the  parts  had  not  been  organized  into  a  compre¬ 
hensive  information  system.  The  new  system  design,  on  the 
other  hand,  used  the  concept  of  a  single  integrated  data  base 
— that  is,  data  would  be  entered  only  once  into  the  system  for 
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use  in  all  subsystems,  thus  eliminating  the  need  for  dupli¬ 
cating  data  in  numerous  nonintegr ated  files.  This  data 
would  be  accessible  directly  from  video  terminals. 

The  Foundation  chose  to  put  all  of  the  functional  parts 
of  the  system  into  operation  at  the  same  time  rather  than 
implementing  them  in  a  series  of  time-phased  steps.  The 
latter  approach  would  have  provided  better  control  of  system 
development  and  quality.  Since  then,  the  system  has  exper¬ 
ienced  numerous  technical  and  management  problems  as  well 
as  resistance  from  agency  personnel. 

In  1976,  a  management  consultant  firm,  D.  P.  Management 
Corporation  of  Lexington,  Massachusetts,  was  hired  to  assess 
the  MIS  project.  The  consultant  compared  MIS's  performance 
with  the  Foundation's  goals  and  concluded  in  January  1977 
that  the: 

— MIS  goals  were  ambitious  but  achievable  over  time; 

— MIS  had  not  yet  met  user  objectives,  although 
progress  had  been  made; 

— efforts  to  maintain  the  system’s  current  program 
and  to  achieve  its  growth  objectives  concurrently 
had  caused  an  imbalance; 

— ADP  administrative  organization  needed  significant 
strengthening;  and 

— many  management  programs  and  procedures  needed  for 
the  successful  operation  and  administration  of  the 
MIS  system  did  not  exist. 

The  consultant  recommended  controlled,  incremental  development 
of  MIS  and  provided  an  action  plan  with  priorities  for  system 
development.  Some  of  these  recommendations  have  been  fully  or 
partially  implemented;  others  have  been  deferred  for  lack  of 
resources. 

System  description 

MIS  uses  a  Honeywell  computer  system  which  provides 
an  integrated  data  base  (JASON)  accessible  from  284  terminals 
throughout  the  Foundation's  headquarters  and  annexes.  MIS 
is  made  up  of  5  major  systems  divided  into  20  subsystems 
with  specific  Foundation  functions  (see  attachment  I). 
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The  major  MIS  systems  are  the: 

— Proposal/Award  Administration  System  which  sup¬ 
ports  the  receipt,  control  and  approval  or  rejection 
of  grant  and  contract  proposals,  and  the  adminis¬ 
tration  of  active  awards; 

— Peer  Review  Administration  System,  which  con¬ 
tains  the  names  of  30,000  potential  reviewers  used 
by  the  Foundation  to  decide  on  grant  proposals; 

— Grantee/Contractor  Information  System,  which  main¬ 
tains  business  and  historical  information  about  insti¬ 
tutions  and  researchers  having  Foundation  grants; 

— Administrative  Management  System,  which  supports 
the  Foundation's  management  and  administration;  and 

— Local  Management  System,  which  gives  services  such 
as  individualized  data  searches,  specialized  report 
formats,  query  of  data  bases  outside  the  Foundation, 
and  development  of  individual  computer  programs. 

Users  of  the  system  include  Foundation  management,  the 
Congress,  the  Office  of  Management  and  Budget,  and  grantee 
institutions.  The  primary  user  is  the  Foundation  itself, 
where  upper  level  managers  use  administrative  and  financial 
data  for  overall  Foundation  management.  Increasingly,  MIS 
is  being  used  by  other  Foundation  personnel,  such  as  program 
managers  and  administrative  staff,  for  monitoring  grant  fund¬ 
ing  levels  and  expenditures,  tracking  proposals  through  pro¬ 
cessing,  and  expediting  letter  writing. 

System  management 

MIS  is  managed  under  the  Directorate  for  Administration 
through  the  Division  of  Information  Systems  (DIS)  which  is 
comprised  of  five  branches: 

— Application  Systems  Branch — responsible  for  all  MIS 
subsystems,  software  development  and  maintenance, 
requirements  analysis,  design  specifications,  and 
software . 

— Co  iputer  Facility  Management  Branch — responsible  for 
the  computer  facility  including  the  terminals,  oper¬ 
ating  system  software,  executive  control,  and  manage¬ 
ment  of  the  integrated  data  base. 
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— Customer  Service  Branch — responsible  for  office  auto¬ 
mation,  directorate  user  representatives,  user  train¬ 
ing,  and  technical  writing. 

— Systems  Support  Services  Branch — responsible  for  input 
services,  processing  support,  fund  controls,  and  the 
documentation  library. 

— Management  Controls  Branch — responsiDle  for  data  qual¬ 
ity,  system  standards,  security,  system  project  manage¬ 
ment,  and  certification  of  new  software. 

The  system  is  operated  by  a  staff  of  123,  of  which  80  are 
Foundation  management  and  administrative  personnel  and  48 
are  contractor  personnel  who  operate  the  computer  hardware 
and  develop  system  software. 

Our  audit  found  that  MIS  users  continue  to  cite  techni¬ 
cal  and  management  problems  affecting  system  reliability  in¬ 
cluding  inaccurate  and  incomplete  data,  slow  response  time  at 
the  computer  terminals,  and  lack  of  attention  to  the  user's 
needs.  To  solve  these  and  other  problems,  improvements  are 
needed  in  five  areas:  quality  control,  system  response  time, 
long-range  planning,  performance  evaluation,  and  system  manage¬ 
ment.  These  areas  are  discussed  in  the  following  text. 

DATA  QUALITY  CONTROL  PROCEDURES  AND 
IMPROVED  SYSTEM  RELIABILITY  NEEDED 


The  Foundation  does  not  have  procedures  to  monitor  data 
quality  and  identify  system  problems  that  cause  poor  data 
quality.  Furthermore,  our  assessment  of  system  reliabilty 
identified  other  areas  which  need  strengthening. 

Data  quality  is  not  monitored 

The  Foundation  has  no  systematic  process  for  monitoring 
data  quality  or  statistically  measuring  the  reliabilty  of  the 
system.  Several  one-time  efforts  have  been  made  to  correct 
selected  portions  of  the  data  base  where  known  errors  existed. 
However,  no  formal  and  regular  effort  has  been  made  to  iden¬ 
tify  the  causes  and  solutions  of  the  problems  to  prevent  recur¬ 
rences.  In  addition,  the  system  lacks  documented  error-cor¬ 
rection  procedures  to  insure  timely  and  appropriate  corrections 
to  errors  that  are  found. 

The  Foundation's  consultant  did  not  study  data  quality 
because  he  found  problems  in  the  system  that  could  contribute 
to  poor  data  quality.  However,  he  recommended  3  years  ago 
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that  the  Foundation  undertake  such  a  study  within  6  months. 
Nevertheless,  the  Foundation  has  not  performed  the  recommended 
data  quality  study  and  has  given  it  low  priority  because 
officials  believe  the  data  quality  is  generally  good.  In 
support  of  their  view,  system  managers  cite  a  1976  internal 
study  of  the  Proposal  and  Application  Information  Subsystem 
which  examined  that  subsystem  during  its  first  full  year  of 
operation.  A  total  of  9,539  observations  were  made  on  41 
data  elements  in  a  sample  of  947  proposals  from  all  Founda¬ 
tion  directorates.  1/  The  result  was  an  overall  accuracy 
of  95.1  percent,  and  98.9  percent  for  critical  data  elements. 

As  part  of  our  reliability  assessment,  we  made  spot 
checks  of  152  critical  data  elements  from  six  proposals  and 
four  awards  to  see  if  computerized  data  and  original  records 
agreed.  Almost  no  errors  were  found.  We  also  made  a  spot 
check  of  duplicate  names  in  the  principal  investigator 
subsystem.  Out  of  2,176  entries  in  the  alphabetical  list 
of  names  for  the  letter  J,  5  percent  possible  duplications 
were  found.  Further  limited  testing  of  seven  data  elements 
in  the  financial  accounting  subsystem  data  from  four  awards 
and  three  proposals  showed  no  errors. 

These  spot  checks  do  not  represent  the  quality  of  data 
in  the  28,000  proposals  or  24,000  awards  in  process  annually, 
but  were  done  to  determine  the  kinds  and  quantity  of  errors 
which  might  be  anticipated  in  a  full-scale  quality  audit. 

Both  the  Foundation's  study  of  1976  and  our  spot  checks 
were  concerned  only  with  the  value  of  basic  data  elements 
at  a  certain  time,  not  with  what  happens  to  data  during 
the  running  of  individual  programs  or  with  the  operation 
of  all  subsystems  over  time. 

Through  a  user  questionnaire  we  solicited  comments 
from  a  selection  of  27  Foundation  division  directors  and 
program  officers  on  the  quality  of  data  in  two  widely- 
used  routine  MIS  reports  and  the  usefulness  of  MIS  in 
carrying  out  their  work.  Program  officers  commented  on  the 
Workload  Status  Report  and  division  directors  on  the  Pro¬ 
posal  Aging  Report.  Respondents  also  provided  numerous 
narrative  comments  on  MIS.  The  results,  which  follow,  are 
the  opinions  of  a  smaller  selection  of  users,  but  are  not 
necessarily  representative  of  the  opinions  of  users  as  a 
whole . 


J^The  Foundation  is  organized  around  the  Office  of  the 

Director,  six  program  directorates,  and  one  administration 
directorate.  (See  attachment  II.) 
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USER  REACTION  TO  MIS  REPORTS— 

•RESPONSES  TO 

KEY  QUESTIONS 

Workload  status 

Proposal  aging 

report 

report 

Is  the  reported  data:  Percent  Yes 

Percent  Yes 

— accurate  and  reliable? 

82 

29 

— complete? 

73 

86 

— available  early  enough? 

67 

79 

— current? 

67 

38 

— useful? 

67 

92 

— understandable? 

100 

100 

Can  the  report  be  used  as  is 
without  further  correcting, 
explanation,  or  analysis? 

69 

57 

Do  you  maintain  manual 
records  to  supplement  com¬ 
puter  produced  information?  1/ 

69 

71 

Does  the  report  duplicate 
any  information  you  now 
receive?  2/ 

25 

71 

On  a  scale  of  1  to  10: 

In  the  work  of  your  office  or 
division,  the  report  is: 

(not  important  at  all  =  1, 
very  important  =  10) 

5 

5 

How  useful  is  the  MIS  to  you 
in  carrying  out  your  duties 
and  responsibilities:  (not 
useful  at  all  =1,  very  use¬ 
ful  =  10) 

5 

6 

1/Some  respondents  said  manual  records  were  kept  because  auto¬ 
mated  records  were  unreliable,  but  others  indicated  that 
manual  records  might  be  kept  anyway  as  a  backup  system. 


2/3ome  respondents  said  duplication  provided  a  useful  cross¬ 
check  of  data  and  aided  in  error  identification. 
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Representative  respondent  comments 

— MIS  seems  to  be  a  closed  system  and  used  primarily 
by  DI3  and  upper  managers.  The  needs  of  the  program 
officials  are  secondary. 

— MIS  data  are  too  unreliable  to  be  of  use,  necessitating 
toe  maintenance  of  manual  records  to  check  the  computer. 

— The  system  is  improving  over  time,  but  still  has  a 
long  way  to  go. 

— DI3  should  find  out  what  programs  personnel  need  be¬ 
fore  generating  software. 

— Too  much  time  is  required  locating  errors  in  the  finan¬ 
cial  accounting  system  and  identifying  such  for  the 
Division  of  Financial  and  Administrative  Management. 

— The  system  will  never  be  better  than  the  accuracy 
and  completeness  of  its  data. 

— Response  time  at  the  terminals  is  often  unacceptable. 

— Down  time  on  the  system  is  much  too  frequent  and  often 
comes  at  the  end  of  the  month  when  it  is  important 
to  get  actions  processed. 

— DIS  staff  are  most  helpful  and  courteous  in  respond¬ 
ing  to  any  questions  we  have. 

The  results  of  the  questionnaire  survey  including  the  nar¬ 
rative  comments  strongly  indicate  the  need  for  improved  system 
reliability . 

System  reliability  needs  strengthening 

Our  reliability  assessment  is  a  judgment  based  on  a 
structured  analysis  of  organizational  and  computer  operation 
controls  and  procedures  including  access  controls,  file  con¬ 
trols,  disaster  recovery,  teleprocessing,  input  and  output 
controls,  and  user  opinions.  We  obtained  information  neces¬ 
sary  for  our  evaluation  from  system  managers  and  users  through 
interviews  and  questionnaires.  (Time  constraints  did  not 
allow  verification  of  this  information  with  actual  Foundation 
procedures  or  system  operations.)  Reliability  is  evaluated  in 
terms  of  risk  (potential  for  error).  High  reliability  indi¬ 
cates  low  risk,  low  reliability  indicates  high  risk. 


,  ,  r  it 
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Our  assessment  revealed  system  strengths  and  weaknesses 
and  resulted  in  a  rating  of  medium  reliability,  which  means 
that  a  user  should  audit  any  system  data  significant  to 
critical  decisionmaking.  A  discussion  of  these  strengths 
and  weaknesses  follows: 

Strengths 

— Steering  committee — Top  management  involvement,  in 
the  form  of  a  steering  committee,  is  essential  to  the 
development  and  operation  of  an  information  system. 

In  July  1977,  the  Foundation  established  the  MIS 
steering  committee  to  provide  top  management  parti¬ 
cipation  in  all  phases  of  MIS  planning  and  resource 
allocation.  However,  the  committee,  composed  of 
assistant  directors,  was  chaired  by  an  assistant 
director  until  January  1979,  when  the  Deputy  Direc¬ 
tor  was  made  chairman.  The  committee  lacks  full¬ 
time  staff  support  needed  to  function  effectively 
and  has  spent  much  of  its  effort  thus  far  defining 
its  role. 

— Separation  of  duties — Separation  of  duties  enhances 
system  security  and,  by  providing  an  effective  check 
against  losses  from  carelessness  or  fraudulent  manip¬ 
ulation  of  data,  promotes  accuracy  and  reliability 
in  data.  Within  DIS  branches,  we  found  different 
Foundation  and  contractor  employees  performing  such 
duties  as  maintenance  of  the  operating  system  and 
data  management  system,  system  design  and  programming, 
hardware  operations,  file  maintenance,  and  data  input. 

— Security — Physical  equipment,  production  programs  and 
procedures,  and  data  files  should  be  adequately 
secured  to  prevent  unauthorized  access.  We  found 
access  to  the  computer  area  and  hardware  is  limited 
to  necessary  personnel  through  the  use  of  key  locks 
and  visitor  sign-in  logs  and  badges.  Account  codes, 
authorization  codes,  and  passwords  are  controlled 
to  prevent  unauthorized  use  of  data  files  and  pro¬ 
cedures.  Other  DIS  security  measures  prevent  the 
unauthorized  entry  of  program  changes  and  data  through 
the  operator's  console,  detect  attempts  at  unauthor¬ 
ized  system  intervention,  and  prevent  unauthorized 
modifications  to  production  programs.  In  addition, 
the  MIS  tape  library  is  accessible  only  to  authorized 
librarians. 

— Disaster  recovery — Disaster  recovery  controls  are  pre¬ 
ventive  processes  that  help  protect  critical  data 
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files,  software,  and  systems  documentation  from  fire 
and  other  hazards,  allowing  a  continuation  of  data 
processing  activities  if  such  hazards  occurred.  DIS 
has  established  such  procedures  describing  the  action 
to  be  taken  in  the  event  of  a  disaster  involving  the 
data  center,  data  files,  and  computer  programs.  Among 
these  are  provisions  for  copying  and  retaining  master 
files.  Sufficient  generations  of  files  are  main¬ 
tained  to  facilitate  reconstruction  of  data  records, 
and  copies  of  operating  and  production  programs  are 
also  maintained.  These  critical  data  files  and  soft¬ 
ware  are  stored  in  a  vault  outside  the  data  center 
and  at  an  offsite  storage  facility. 

— Other  controls  present — Traditional  control  concepts 
are  augmented  by  controls  written  into  computer  pro¬ 
grams  and  other  system  software  as  well  as  controls 
built  into  the  computer  hardware.  The  MIS  executive 
control  is  a  collection  of  software  modules  which 
control  access  to  the  integrated  data  base  and 
journalize  all  transactions  which  alter  the  data 
base.  Journalization,  the  storage  of  before  and 
after  images  of  all  altered  pages  in  the  data  base, 
provides  an  audit  trail  for  analysis  of  data  base 
integrity  and  system  restoration.  In  addition, 
computer  programs  include  various  types  of  edits 
and  routines  which  are  helpful  in  assuring  that  data 
is  properly  processed  through  the  entire  run. 

Weaknesses 


— Internal  audit  oversight — Internal  audit  oversight  of 
data  processing  activities  helps  to  insure  that  com¬ 
puter-produced  information  is  accurate  and  reliable. 
However,  the  Foundation's  audit  office  has  had  little 
involvement  in  DIS  activities.  Internal  audit  moni¬ 
toring  has  consisted  of  minimal  participation  in  MIS 
design  and  implementation,  membership  in  DIS  study 
team  efforts,  and  verification  of  information  on 
computer  reports.  Although  an  operational  audit  of 
MIS  has  been  a  part  of  the  yearly  audit  plan  since 
FY  1974,  the  internal  audit  staff  has  not  been  able 
to  perform  a  detailed  review  due  to  limited  funds, 
time,  and  technical  expertise. 

— System  documentation — Internal  control  in  an  auto¬ 
matic  data  processing  department  requires  adequate 
documentation  describing  both  the  system  (including 
the  application  systems  and  programs)  and  the  proce¬ 
dures  used.  However,  DIS  management  has  not  encouraged 
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the  systems  analysts/pr ogr ammer s  to  maintain  and  up¬ 
date  systems  documentation,  feeling  it  would  divert 
their  limited  resources  from  system/program  main¬ 
tenance  and  other  more  important  functions.  The 
amount  of  documentation  varies  for  each  system  (sub¬ 
system).  Original  system  documentation  which  was 
developed  by  the  contractor  and  turned  over  to  the 
Foundation  during  implementation  has  not  been  updated 
or,  in  some  cases,  retained.  Although  these  systems/ 
programs  have  undergone  considerable  testing,  test 
plans  and  their  results  are  not  available.  System 
descriptions  and  flowcharts  are  developed  when  needed 
for  documents  and  briefings.  In  addition  to  the  lack 
of  systems  documentation,  we  observed  limited  proce¬ 
dural  documentation.  For  example,  DIS  could  not  pro¬ 
vide  documentation  for  ADP  personnel  duties  and  respon¬ 
sibilities,  other  than  position  descriptions.  Although 
tape  library  procedures,  disaster  recovery  procedures, 
and  security  procedures  are  documented,  error  correc¬ 
tion  procedures  are  not.  This  can  result  in  standard 
procedures  not  being  followed  and  errors  not  being 
corrected  promptly  and  appropriately. 

— System  utilization  data — Utilization  data  is  essential 
to  management  for  measuring  the  efficiency  and  effec¬ 
tiveness  of  tne  data-processing  system.  Available 
utilization  data  includes  the  names  of  individuals 
accessing  MIS,  the  number  of  times  they  were  on 
line  per  period,  and  total  lapsed  time  for  each  indi¬ 
vidual  per  period;  for  batch  processing,  the  number 
of  jobs  run  daily.  No  data  is  gathered  which  could 
be  used  to  determine  the  number  of  times  each  proce¬ 
dure  was  accessed  per  period,  the  number  of  job  reruns, 
or  total  processing  time  for  each  program.  Failure 
to  obtain  and  analyze  these  types  of  information  could 
result  in  inefficient  utilization  of  MIS . 

— Internal  controls — Internal  controls  safeguard  re¬ 
sources  from  waste,  fraud,  and  inefficiency  and  pro¬ 
mote  accuracy  and  reliability  in  accounting  and 
operating  data.  Although  MIS  functions  appear  to 
be  secure,  several  procedures  need  to  be  improved. 

For  example,  tighter  controls  are  needed  over  system 
and  program  documentation  kept  at  the  system  analysts' 
and  programmers'  desks  to  prevent  unauthorized  access  to 
such  material.  Also,  sensitive  output,  such  as  personal 
information,  is  discarded  along  with  nonsensitive 
information  instead  of  being  shredded  or  burned. 

There  is  no  nonprinting/nondisplaying  or  obliteration 
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facility  incorporated  in  the  terminals  to  prevent 
unauthorized  persons  from  having  access  to  accounting 
and  authorization  codes  and  passwords.  These  addi¬ 
tional  control  procedures  would  help  produce  a  more 
secure  climate  for  data  processing. 

System  design  problems 

In  addition  to  the  security  and  system  control  weaknesses, 
we  learned  about  system  design  problems  in  the  Principal  In¬ 
vestigator/Project  Director  { PI/PD)  Subsystem.  The  MIS  inte¬ 
grated  data  base  allows  data  to  be  strung  out  in  chains,  allow¬ 
ing  records  to  be  linked  by  pointers  within  each  record  and 
permitting  the  linkage  of  related  files  so  that  data  need 
be  entered  only  once.  However,  this  chain  linkage  is  absent 
between  the  PI/PD  Subsystem  and  the  integrated  data  base. 
Instead,  this  subsystem  is  linked  by  common  data  elements 
within  the  integrated  data  base.  Any  changes  to  the  PI/PD 
data  records  must  be  made  individually  to  each  common  data 
element.  Errors  could  be  made  while  making  these  changes  or 
some  common  data  elements  could  be  overlooked,  creating 
unsynchronized  data  within  the  data  base.  Another  problem, 
the  absence  of  a  standard  identifier  within  PI/PD  records, 
can  lead  to  duplicate  entries  in  the  data  base.  DIS  is  cur¬ 
rently  seeking  solutions  to  both  of  these  problems. 

Our  data  quality  and  reliability  assessments  were  per¬ 
formed  from  October  through  November  1978  and  apply  only  to 
that  time  frame.  The  results  are  not  necessarily  typical  of 
the  system  over  other  time  frames  because  the  system  is 
changing  with  respect  to  design,  procedures,  and  management 
controls,  and  there  is  no  ongoing  data  quality  assessment 
process.  The  Foundation  has  system  and  computer  program 
change  control  procedures  that  provide  some  control  over,  and 
a  chronological  history  for,  the  modifications  made  to  the 
system. 

IMPROVEMENT  IN  SYSTEM  RESPONSE  TIME  NEEDED 

System  users  and  managers  reported  experiencing  exces¬ 
sive  response  times  at  the  terminals,  sometimes  as  long  as 
5  minutes.  System  managers  want  a  response  time  of  under 
5  seconds.  The  problem  was  generally  described  as  an  over¬ 
load  problem  arising  when  20  or  more  of  the  Foundation's 
238  (at  the  time  of  our  study)  MIS  terminals  were  concur¬ 
rently  active.  The  Foundation's  consultant  noted  in  his  re¬ 
port  that  response  times  of  several  minutes  were  common.  We, 
too,  experienced  long  delays  in  getting  on  the  system  and 
subsequently  retrieving  data  during  information  gathering 
for  our  audit. 
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While  system  managers  agreed  that  a  response  time 
problem  had  existed  for  at  least  2  years,  they  had  not  de¬ 
fined  the  exact  degree  of  the  problem  or  identified  its 
causes.  Instead,  they  have  been  trying  to  work  around  the 
problem  by  requesting  program  personnel  to  use  MIS  at  times 
when  the  computer  center  was  not  performing  high  volume  data 
entry.  System  managers  were  looking  to  a  study  by  the  Federal 
Computer  Performance  Evaluation  and  Simulation  Center  (FEDSIM) 
to  assess  the  problem  further. 

In  February,  1979,  FEDSIM  began  studying  the  Foundation’s 
computer  system  analyzing  workload  characteristics,  system 
response,  system  design,  system  utilization  levels,  and 
operational  procedures.  The  study  included  two  system  load¬ 
ing  experiments  to  determine  the  number  of  users  that  could 
be  adequately  serviced  on  the  present  computer,  and  to  iden¬ 
tify  potential  system  bottlenecks  during  peak  use  periods. 

The  principal  conclusion  of  the  FEDSIM  study  was  that  signi¬ 
ficant  improvements  in  slow  response  times  were  not  to  be 
found  by  adding  central  system  hardware.  Rather,  they  would 
come  from  better  management  of  existing  computer  resources. 

The  FEDSIM  findings  were  qualified  by  the  instrumentation 
available  and  the  study  methodology  used.  For  example,  JASON, 
one  of  the  two  major  systems  tested,  lacked  the  software 
instrumentation  necessary  to  collect  terminal  response  times. 
Since  response  time  delays  had  been  observed  using  JASON, 
this  limited  the  study's  ability  to  identify  and  suggest 
specific  solutions  to  response  time  problems.  Other  qualifi¬ 
cations  include  a  limited  workload  and  operational  environ¬ 
ment  measured  under  controlled  conditions;  no  batch  pro¬ 
cessing  load  applied  during  the  system  loading  experiments; 
and  Foundation  staff  and  representative  workloads  used  in 
the  experiments  were  selected  and  coordinated  by  Foundation 
personnel,  not  FEDSIM.  (See  Agency  Comments  and  Our  Evalu¬ 
ation  on  page  23  for  additional  comments.) 

FEDSIM  reported  that  those  system  response  times  that 
were  measured  were  generally  good  during  the  testing.  (Most 
were  in  a  range  of  1  to  10  seconds.)  However,  some  delays 
ranging  from  6  seconds  to  6  minutes  and  15  seconds  occurred. 
The  tests  also  demonstrated  that  up  to  71  users  could  access 
the  system  concurrently  under  the  controlled  conditions  of 
the  test. 

The  study  reported  that  the  Foundation  does  not  have 
adequate  system  accountability  data  for  assessing  and  manag¬ 
ing  the  performance  of  its  computer  system  under  varying 
workloads.  Improving  this  condition  was  FEDSIM's  first 
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recommendation  to  the  Foundation.  Additional  FEDSIM  findings 
affecting  system  performance  and  responsiveness  include: 

— Competition  exists  between  data  entry  personnel  and 
system  users  for  computer  resources. 

— Formal  response  and  turnaround  time  requirements  for 
system  users  do  not  exist,  together  with  lack  of 
communication  with  users  over  what  constitutes  rea¬ 
sonable  expectations  of  computer  performance. 

— Most  batch  workloads  are  processed  during  the  day 
shift  when  users'  demands  are  at  a  peak. 

— Central  processing  unit  and  memory  resources  are  al¬ 
most  fully  used  during  the  day  shift. 

— System  resources  are  not  a  constraint  to  additional 
workload  during  evening,  night,  and  weekend  processing 
periods. 

FEDSIM  recommended  many  changes  to  the  system  including 
the  following: 

— Enhance  the  policies  and  procedures  for  system  account¬ 
ing  to  provide  Foundation  personnel  with  better  in¬ 
formation  by  which  to  manage  computer  resources. 

— Establish  performance  goals  for  the  system  by  identify¬ 
ing  formal  response  and  turnaround  requirements  for 
system  users. 

— Reevaluate  the  design  of  the  Office  (Local)  Management 
System  to  meet  user  needs  while  reducing  load  on  the 
system,  and 

— Improve  guidance  to  users  concerning  th  optimum  use 
of  the  system. 

Even  though  the  Foundation  has  no  continuing  performance 
evaluation  program,  it  has  continued  to  expand  the  number  of 
terminals  accessing  the  system  without  knowing  whether  this 
expansion  will  further  delay  system  response  time.  In  1977  the 
Foundation  added  a  second  communications  control  unit  with  16 
communications  lines  to  the  MIS  computer,  which  already  had 
64  lines.  The  Foundation  added  46  additional  terminals  in 
fiscal  year  1979  to  the  238  it  already  had. 
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LONG-RANGE  PLANNING  NEEDED 

The  Foundation's  MIS  management  program  lacks  both  a 
long-range  plan  and  a  planning  process  to  establish  quanti¬ 
fied  goals,  priorities,  and  target  dates.  As  early  as  1972, 
Foundation  management  recognized  that  past  efforts  to  auto¬ 
mate  Foundation  files  had  been  too  narrow  in  scope  and  had 
led  to  disappointing  results.  While  development  of  MIS 
was  based  on  a  5-year  plan,  the  plan  ended  in  1976  with  no 
continuing  long-range  planning  process  in  existence.  The 
Foundation  then  contracted  for  a  system  evaluation.  The  con¬ 
sultant  cited  the  absence  of  effective  long-range  planning  as 
adversely  affecting  system  management  and  recommended  that  the 
Foundation  immediately  develop  a  long-range  plan  and  a  plan¬ 
ning  process.  He  emphasized  the  need  for  long-range  hardware 
planning  to  control  costs  and  efficiency  of  operations  and 
recommended  that  a  long-range  hardware  plan  and  a  computer 
optimization  study  be  done  immediately.  He  also  recommended 
the  immediate  implementation  of  a  system  development  manage¬ 
ment  program  which  would  provide  for  the  orderly  development 
of  system  software. 

After  3  years,  the  Foundation  still  has  not  developed 
these  long-range  plans,  performed  the  optimization  study, 
or  implemented  a  system  development  management  program.  DIS 
officials  said  these  projects  have  received  low  priority 
because  daily  operation  and  maintenance  of  the  system  was 
consuming  the  available  resources.  Furthermore,  no  firm 
plans  exist  for  pursuing  these  projects. 

Currently  MIS  is  managed  on  a  short-range  basis, 
essentially  reacting  to  daily  demands  and  problems.  While 
numerous  work  projects  for  system  improvements  have  been 
identified,  there  were  often  no  detailed  documented  plans 
for  the  improvements  nor  funds  to  put  them  into  effect. 

IMPROVEMENT  III  PERFORMANCE  EVALUATION  NEEDED 

Forty-thi ee  Foundation  personnel  including  upper-level 
managers,  program  officers,  and  administrative  and  financial 
personnel  answered  questionnaires.  The  users  stated  that 
MIS  should  perform  better  than  it  does  and  that  greater 
attention  is  needed  to  determine  and  meet  user  information 
needs.  These  issues  are  discussed  below. 

Users  expect  MIS  to  perform  better 

Most  users  criticized  MIS  for  unreliability  in  data 
accuracy  and  completeness  and,  as  previously  discussed,  for 
unsatisfactory  delays  in  getting  on  the  system  and  long  waits 
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at  the  terminals  for  data  access.  3ecause  of  such  problems 
many  users  have  kept  in  their  offices  manual  records  which  are 
used  for  computerized  data  verification  and  for  program  manage¬ 
ment.  Several  users  noted  that  DIS  had  asked  program  officers 
to  review  long  computerized  lists  of  proposals  and  awards  for 
their  divisions  at  the  end  of  fiscal  year  1978  to  help  correct 
errors,  reconcile  financial  data,  and  find  proposals  and 
awards  which  had  become  lost  in  the  system.  One  user  crit¬ 
icized  the  system  for  not  having  established  error  correction 
procedures  so  users  could  clearly  describe  the  error  discovered 
and  recommend  how  it  should  be  corrected.  Another  user  stated, 
however,  that  the  computer  was  not  to  blame  for  all  errors  be¬ 
cause  some  were  caused  by  the  program  office  staff's  failing 
to  identify  and  report  errors. 

A  better  assessment  of  user  needs  is  required 

*fe  found  no  adequate  process  for  evaluating  and  address¬ 
ing  user  needs.  Except  for  the  initial  requirements  study  of 
1971,  the  Foundation  has  not  systematically  assessed  user  needs. 
In  answering  our  questionnaires  (see  p.  7)  users  said  that 
MIS  is  only  moderately  important  and  useful  in  helping  them 
carry  out  tneir  duties.  They  identified  problems  which  indi¬ 
cate  that  MIS  is  not  adequately  addressing  their  needs. 

For  example,  users  said  that  MIS  reports  required  further  cor¬ 
rection,  explanation,  or  analysis,  that  manual  records  were 
needed  to  supplement  computer-produced  information,  and  that 
some  reports  duplicated  information  already  received. 

Some  program  officers  complained  that  some  routine  MIS 
reports  were  not  useful  and  therefore  not  used.  Instead,  per¬ 
sonnel  used  their  own  computer  programs  to  develop  special- 
format  reports  tailored  to  their  specific  needs.  Other  criti¬ 
cisms  included  heavy  use  of  numerical  codes  for  classifying 
data,  which  required  frequent  use  of  a  code  dictionary,  the 
undesirable  format  of  data,  lack  of  effective  training  for  use 
of  the  system,  and  inadequate  communication  on  user  needs  be¬ 
tween  system  users  and  system  managers. 

The  Division  of  Information  Systems  has  assumed  the  task 
of  determining  the  current  needs  of  program  users.  In  doing 
so,  system  managers  have  used  feedback  from  program  personnel 
through  user  representatives  (DIS  personnel  located  within 
Foundation  directorates)  and  the  directorate  task  groups 
(committees  that  monitor  MIS  performance).  These  channels 
are  still  the  main  sources  for  identifying  user  needs,  since 
the  Foundation  has  not  established  a  performance  evaluation 
program  to  assess  user  needs  and  how  well  they  are  served. 

The  consultant  recommended  such  a  program  3  years  ago. 
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STATUS  OF  OVERALL  SYSTEM  MANAGEMENT 


The  consultant  made  23  recommendations  for  system  improve¬ 
ments  in  applications  organization,  and  systems  and  resources 
management  programs.  Of  these  the  Foundation  has  addressed 
13  recommendations,  partially  addressed  2,  and  not  addressed 

3. 


For  those  addressed  we  did  not  assess  the  extent  to  which 
the  changes  have  effectively  improved  MIS  performance.  How¬ 
ever,  a  description  of  the  recommendations  and  the  status  of 
implementation  is  presented  below.  (See  attachment  III  for 
additional  details.) 

Recommendations  addressed 


1.  Establish  a  hierarchy  of  steering  committees.  Two 
committees  were  recommended:  one,  composed  of 
upper  level  managers,  would  determine  overall 
policy,  funding,  goals,  and  objectives;  the 
second,  composed  of  operational  staff,  would 
monitor  policy  and  program  implementation. 

The  first  committee  was  established  (July  1977), 
but  not  the  second.  Instead  directorate  task 
groups  were  established  in  early  1978  to 
monitor  MIS  implementation. 

2.  Provide  user  representatives.  Each  directorate 
now  has  a  program  analyst  from  DIS  located  on¬ 
site  to  help  program  personnel  use  MIS.  User 
reaction  was  favorable  in  GAO's  questionnaire 
responses . 

3 .  Establish  a  "quick  response"  service  group.  A 
staff  of  four  has  been  established  in  DIS  to  ex¬ 
pedite  MIS  response  to  external  requests  for  data. 

4 .  Develop  an  implementation  plan  for  a  financial 
accounting  system.  The  consultant  developed  a 
plan  which  the  Foundation  is  now  implementing 
on  a  5-year  basis.  Implementation  is  now  in 
its  third  year. 

5.  Establish  a  DIS  administrative  staff.  Quality 
control  of  MIS  products  and  activities  has  been 
fragmented  among  numerous  parties  with  none  hav¬ 
ing  authority  to  establish  standards.  As  a  reme¬ 
dy,  DIS  established  a  management  controls  branch 
(April  1977)  responsible  for  quality  control,  system 
development,  cost  accounting,  training,  and  security. 
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Thus  fac  the  branch's  activities  have  been  limited 
to  organizing  itself  and  performing  ad  hoc  tasks. 
Resource  limitations  reportedly  have  prevented 
greater  achievements. 

6.  Enhance  award  processing.  The  consultant  stated 
that  the  Foundation  could  enhance  its  award  process¬ 
ing  by  automating  award  letters  and  institutional 
indirect  cost  rates.  Both  capabilities  have  been 
added.  Also,  the  financial  accounting  system  now 
automatically  reflects  award  actions. 

7.  Enhance  award  management.  The  consultant  stated 
that  control  and  monitoring  of  active  awards  was 
inefficient  because  of  numerous  manual  operations 
which  could  be  automated;  for  example,  tickler 
filer  used  to  identify  delinquent  reports.  The 
Foundation  has  made  the  recommended  enhancement 
and  is  currently  reviewing  and  redesigning,  where 
necessary,  the  Awards  Management  System. 

8.  Conduct  requirements  study.  The  consultant  per¬ 
formed  a  requirements  study  for  the  Foundation  in 
early  1977  which  assessed  detailed  work  projects, 
priorities,  and  resources  required.  The  study  was 
to  be  the  initial  step  in  developing  a  long-range 
planning  process.  The  study  concluded  that  the  MIS 
workload  was  well  in  excess  of  available  resources, 
thus  requiring  critical  management  decisions. 

9.  Centralize  ADP  expenditures.  Several  directorates 
contract  for  supplementary  data  processing  services 
tailored  to  specific  needs  (cost  is  approximately 
$500,000  annually).  Previously  these  contracts  were 
not  coordinated  with  DI3.  Now  the  steering  commit¬ 
tee  reviews  and  approves  these  expenditures. 

10.  Implement  certain  office  automation  modules.  Office 
automation  is  the  use  of  computers  to  perform  office 
processes  otherwise  done  manually.  MIS  provides 
programs  (modules)  for  such  processes  as  letter 
writing,  text  editing,  data  base  searches,  and  dic¬ 
tation  services.  Twenty  percent  of  the  Foundation's 
divisions  have  some  modules  installed. 

11.  Defer  automation  of  general  office  functions.  Con¬ 
ceivably  office  automation  can  result  in  using 
electronic  media  totally,  instead  of  paper.  The 
Foundation  has  limited  office  automation  to  word¬ 
processing  applications  and  deferred  automating 
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processes  that  would  eliminate  altogether  the  use 
of  paper. 

12.  Develop  a  skills  improvement  program.  The  consultant 
noted  that  DI3  needed  to  develop  better  managerial 
and  ADP  technical  skills  in  its  staff.  Meanwhile, 

it  recommended  contracting  for  software  until  in- 
house  skills  could  be  improved.  Since  then,  DIS  has 
contracted  for  all  major  software.  Formal  training 
courses  have  been  identified  and  made  available  to 
personnel . 

13.  Make  miscellaneous  enhancements.  Numerous  detailed 
and  technical  recommendations  were  made  for  strength¬ 
ening  major  subsystems  by  1930.  The  Foundation  has 
followed  up  by  identifying  numerous  subsystem  pro¬ 
jects,  many  of  which  were  completed  in  FY  '78.  An 
additional  80  projects  remain,  38  of  which  were 
scheduled  for  FY  '79.  The  remaining  42  projects  are 
unfunded  and  include  such  tasks  as  user  documentation 
of  system  procedures,  evaluation  and  enhancement  of 
subsystems,  providing  report  numbers  on  recurring 
reports,  testing  annual  closing  procedures  for  the 
financial  accounting  subsystem,  and  rewriting  reports 
for  optimization. 

Recommendations  partially  addressed 

1.  Conduct  a  workload  process  study.  The  consultant 
said  a  study  of  workload  processing  procedures 
should  be  done  within  6  months  to  streamline  exist¬ 
ing  procedures.  Subsequently  the  Foundation  had 
the  consultant  perform  such  a  study  in  one  direc¬ 
torate  (science  education).  Studies  in  the  other 
directorates  have  been  deferred. 

2 .  Adopt  a  project  management  organization  structure. 

MIS  development  efforts  have  been  accomplished  by 
numerous  people  working  on  specific  tasks  with  no 
one  individual  responsible  for  overall  coordination. 
The  consultant  said  this  approach  was  hindering  ef¬ 
fective  management  and  recommended  immediate  adop¬ 
tion  of  a  project  manager  approach  where  one  person 
was  responsible  for  specifically  identified  project 
DIS  adopted  the  project  management  principle  within 
its  existing  organizational  units  without  changing 
its  organizational  structure,  on  the  premise  that 
most  MIS  work  is  now  maintenance  and  not  develop¬ 
mental.  Also,  limited  resources  did  not  permit 
a  staff  of  the  size  project  management  requires. 
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Recommendations  not  addressed 


1.  Conduct  data  quality  study.  The  consultant  did  not 
perform  a  data  quality  study  but  recommended  that 
the  Foundation  perform  one  within  6  months.  DIS 
analyzed  and  corrected  errors  in  selected  portions 
of  >-he  data  base  but  did  not  perform  a  major  study 
because  data  quality  was  considered  good.  In  ad¬ 
dition,  resources  were  lacking.  As  we  began  our 
review  DIS  officials  said  they  were  reinstituting 
this  study  with  newly  acquired  resources. 

2 .  Develop  and  maintain  a  long-range  systems  plan. 

The  consultant  cited  the  absence  of  effective  long- 
range  planning  as  adversely  affecting  system  manage¬ 
ment,  and  recommended  that  the  Foundation  pursue 
immediately  a  long-range  plan  and  a  planning  process 
No  long-range  planning  process  or  plan  has  been 
developed.  Currently  DIS  relies  on  the  MIS  steer¬ 
ing  committee  and  the  Management  Controls  Branch  for 
planning . 

3.  Prepare  a  long-range  hardware  plan.  The  consultant 
noted  the  necessity  for  effective  long-range  hard¬ 
ware  planning  in  controlling  costs  and  efficiency 
of  operations  and  noted  that  the  Foundation  was 

not  doing  such  planning.  The  consultant  recommended 
that  a  long-range  hardware  plan  and  a  computer  opti¬ 
mization  study  be  done  immediately.  To  date,  lack 
of  resources  and  low  priority  have  prevented  such 
studies,  according  to  agency  officials. 

4.  Develop  a  performance  management  program.  The  con¬ 
sultant  found  need  for  a  better  ability  to  determine 
and  control  how  well  MIS  is  performing  its  mission 
An  intermediate-level  study  was  recommended  using 
readily  available  data.  DIS  has  not  decided  how  it 
will  act  on  this  recommendation. 

5 .  Develop  a  program-budgeting  and  cost-accounting 
system.  The  Foundation's  efforts  at  ADP  budgeting 
and  cost  accounting  were  judged  as  very  rudimentary, 
and  a  more  formal  systematic  approach  was  recom¬ 
mended.  The  Foundation  has  not  pursued  this  recom¬ 
mendation  and  is  undecided  about  its  ultimate  dis¬ 
position. 

6 .  Develop  and  install  a  systems  development  manage¬ 
ment  program.  DIS  has  lacked  a  systems  development 
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management  program  and  has  relied  on  contractors  to 
develop  its  software.  The  consultant  recommended 
immediate  development  and  installation  of  a  formal 
program  for  this  purpose.  Subsequently,  the 
Foundation  had  the  consultant  develop  a  program, 
but  it  has  not  been  implemented  and  is  not  scheduled 
for  implementation. 

7 .  Reposition  system  analysts  into  the  development 
process .  The  consultant  noted  the  desirability  of 
positioning  system  analysts  between  users  and  pro¬ 
grammers  to  help  insure  that  user  needs  were  ef¬ 
fectively  met.  DIS  has  not  found  it  possible  to  do 
this  because  of  manpower  limitations.  Therefore,  it 
is  seeking  to  develop  system  analyst  skills  within 
its  programmer  ranks. 

8.  Develop  a  new  personnel  system.  The  consultant  found 
the  agency's  personnel  system  needed  major  improve¬ 
ments.  The  Foundation  made  several  enhancements  of 
the  system  pending  a  system  requirements  study. 

SYSTEM  COSTS 

According  to  Foundation  figures,  MIS  has  cost  over  $15 
million  from  fiscal  year  1971  through  fiscal  year  1979, 
excluding  personnel  costs.  Items  of  cost  include  equipment 
rentals  and  purchases,  supplies,  and  software  and  system  de¬ 
velopment  contracts.  Costs  for  fiscal  years  1976-1979  in¬ 
cluding  personnel  are  shown  below.  (See  attachment  IV  for 
greater  cost  detail.) 


Fiscal  Year  ADP  Costs*  Personnel  Total 


(millions) 


1976 

$2.4 

$1.2 

$3.6 

1977 

2.0 

1.4 

3.4 

1978 

2.3 

1.6 

3.9 

1979 

3.8 

1.8 

5.6 

* ADP  costs  include  equipment  rentals  and  purchases,  supplies, 
and  support  contracts. 

During  fiscal  year  1979,  the  Foundation  purchased  the 
second  processor  it  had  been  renting,  but  shelved  plans  to 
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add  a  third  communications  control  unit.  The  Foundation 
added  46  more  computer  terminals. 

CONCLUSIONS 


Significant  improvements  in  MIS  and  its  management  are 
needed  if  the  system  is  to  fulfill  its  intended  purpose  and 
achieve  a  degree  of  reliability  satisfactory  to  its  users. 
Although  3  years  have  passed  since  the  consultant's  evaluation 
study,  the  Foundation  has  not  yet  implemented  major  recom¬ 
mendations  which  the  consultant  said  were  needed  immediately. 
These  recommendations  addressed  data  quality  assessment,  long- 
range  planning,  performance  management,  and  system  management. 

Serious  technical  and  management  problems  threaten  the 
successful  operation  of  the  system.  While  system  managers 
cite  lack  of  resources  as  the  main  reason  for  not  implementing 
important  recommendations,  we  believe  that  decisions  made 
without  long-range  planning  or  knowledge  of  system  effective¬ 
ness  are  also  major  causes.  We  believe  the  Foundation  has 
given  low  priority  to  areas  which  should  have  had  the  highest 
priority.  We  found,  for  example,  that  the  Foundation  is 
giving  higher  priority  to  implementing  office  automation 
than  to  achieving  a  highly  reliable  system.  Also,  the  Foun¬ 
dation  has  added  equipment  to  the  system's  operating  hardware, 
which  is  already  experiencing  response  time  delays,  without 
knowing  what  adverse  impact  this  equipment  will  have  on 
the  system's  overall  performance. 

Additional  equipment  would  serve  to  increase  potential 
demand  on  the  system  during  peak  daytime  hours.  Consider¬ 
ing  the  absence  of  good  system  accountability  data  and 
FEDSIM's  conclusion  that  system  hardware  is  not  a  constraint 
to  improved  system  performance,  we  believe  the  Foundation 
should  carefully  assess  any  further  plans  for  adding  termi¬ 
nals  and  associated  equipment  to  the  system.  Better  manage¬ 
ment  of  existing  computer  resources,  as  recommended  by  the 
consultant  and  FEDSIM,  would  seem  to  be  a  more  desirable 
alternative  at  this  time.  Any  additional  equipment  should 
broaden  the  scope  of  services  to  users,  yet  the  Foundation 
has  not  yet  assessed  how  effective  existing  services  are, 
or  how  such  services  can  be  improved. 

In  our  view,  MIS's  value  to  the  Foundation  and  user  ac¬ 
ceptance  depend  on  high  system  reliabilty.  Our  reliability 
assessment  shows  the  system  to  be  of  medium  reliability, 
which  means  that  users  of  system  data  should  verify  the  ac¬ 
curacy  of  the  data  before  using  it  for  important  decisions. 
System  users  within  the  Foundation,  the  Congress,  the  Office 
of  Management  and  Budget,  and  others  will  not,  nor  should 
they,  be  satisfied  with  data  from  a  medium-reliability  system. 
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Some  changes  have  been  made  to  MIS  as  a  result  of  the 
consultant's  recommendations  but  the  Foundation  has  not 
examined  the  impact  of  these  changes  on  the  system.  So 
far,  improvements  are  not  obvious  since  users  continue  to  cite 
system  unreliability  and  slow  response  time  at  the  terminals. 
They  also  report  that  manual  records  must  be  kept  to  verify 
and  correct  computerized  data  and  that  MIS  reports  are  not 
effectively  meeting  their  needs.  In  addition  they  report  the 
absence  of  good  training  and  documentation  on  using  the 
system  and  correcting  data  errors. 

While  MIS  is  not  yet  fully  implemented.  Foundation 
management  already  depends  on  it  to  provide  many  essential 
services  and  data.  Yet  the  system  will  be  unable  to  provide 
highly  reliable  data  until  the  technical  and  management  prob¬ 
lems  identified  in  this  report  are  addressed  and  remedied. 

In  addressing  these  problems,  we  believe  the  Foundation  should 
reexamine  the  purpose  of  MIS,  clearly  define  its  users  and 
their  needs,  and  establish  specific,  quantified  goals  and  tar¬ 
get  dates  to  correct  deficiencies  and  complete  the  system. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 

We  proposed  that  the  Foundation  take  the  following 
actions  to  improve  MIS:  (1)  resolve  the  slow  system  response 
time  problem,  (2)  assess  MIS's  data  quality  and  establish 
procedures  to  periodically  test  data  quality,  (3)  develop  and 
implement  a  long  range  system  and  hardware  planning  process, 
(4)  conduct  MIS  performance  evaluation  to  include  assessing 
user  needs,  and  (5)  improve  overall  MIS  management  in  areas 
such  as  security  and  training. 

In  a  letter  dated  June  15,  1979  (see  attachment  V),  the 
Foundation  advised  us  that  although  it  had  concerns  about 
our  reliance  on  the  use  of  questionnaires  and  interviews  to 
assess  user  satisfaction  with  MIS,  it  generally  agreed  with 
most  of  our  proposed  recommendations.  The  Foundation  stated 
that  it  was  or  will  be  taking  actions  to  implement  most  of 
our  recommendations  to  improve  MIS.  However,  the  Founda¬ 
tion  did  not  agree  that  it  had  a  slow  system  response  time 
problem  of  the  magnitude  we  described  in  the  draft  report. 

The  Foundation  referred  to  FEDSIM's  study  of  MIS’s  user  re¬ 
sponse  time  which,  the  Foundation  said,  showed  response  time 
to  be  in  an  acceptable  range.  The  Foundation  agreed,  how¬ 
ever,  to  conduct  computer  performance  evaluations  on  a 
regular  basis  to  maintain  an  optimized  system.  Our  evalua¬ 
tion  of  the  specific  points  the  Foundation  raised  in  its 
letter  follows. 
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Audit  methodology  to  assess  user  satisfaction 

The  Foundation  stated  it  had  concerns  regarding  our 
report's  usefulness  because  of  our  reliance  on  interviews 
and  questionnaire  data  to  help  assess  user  satisfaction  with 
MIS.  The  Foundation  noted  that  the  use  of  such  data  could 
make  it  difficult  to  distinguish  between  legitimate  system 
proolems  anc!  other  factors  that  influence  user  satisfaction. 

The  Foundation  suggested  that  our  user  satisfaction  findings 
regarding  two  widely  used  reports  produced  by  MIS  could  not 
be  attributed  solely  to  the  quality  of  MIS's  integrated  data 
base,  but  rather  may  have  been  the  result  of  "differences  in 
user  perception"  of  the  reports. 

Our  questionnaires  and  interviews  identified  a  legiti¬ 
mate  system  problam--er r or s  in  the  data  base — which  appears 
to  adversely  affect  the  accuracy  of  a  widely  used  report  pro¬ 
duced  from  the  data  base.  We  selected  two  widely  used  reports 
produced  by  MIS,  each  of  which  is  used  by  a  different  Founda¬ 
tion  management  group.  Program  managers  use  the  Workload 
Status  Report,  whereas  division  directors  use  the  Proposal 
Aging  Report.  We  asked  only  program  managers  about  the  Work¬ 
load  Status  Report,  while  only  division  directors  were  asked 
about  the  Proposal  Aging  Report.  The  differences  in  user 
perception  appear  very  much  dependent  on  the  data  base  errors 
since  the  Foundation  states  that,  of  the  two  reports  we  tested, 
errors  tend  to  be  concentrated  in  the  report  (Proposal  Aging 
Report)  our  work  showed  had  the  lowest  user  satisfaction  rate 
(29  percent).  Our  questionnaire  showed  that  users  of  the 
Proposal  Aging  Report  (division  directors)  were  having  problems 
with  the  report  because  of  errors,  and  the  Foundation  confirmed 
this  by  noting  that  of  the  two  reports  we  tested,  errors  are 
concentrated  in  the  Proposal  Aging  Report. 

The  Foundation  suggested  in  its  letter  that  although 
errors  are  present  in  the  Workload  Status  Report,  the  errors 
are  not  as  easily  recognized  as  in  the  Proposal  Aging  Report 
because  of  the  larger  volume  of  information  present  in  the 
workload  report.  However,  errors  in  the  data  base  are  still 
errors  no  matter  how  unrecognizable  they  might  be  in  some 
reports.  Errors  are  present  in  some  data  in  the  integrated 
data  base  that  can  adversely  affect  the  accuracy,  and  thus 
the  reliability,  of  a  report  produced  from  the  data  base.  The 
extent  to  which  the  errors  adversely  impact  report  quality 
will  depend  on  the  specific  data  used  to  develop  the  report 
and  the  intended  use  of  the  report.  Merely  suggesting  that 
errors  in  the  data  base  are  just  perceptions  because  the 
errors  are  more  recognizable  in  one  report  than  another 
attempts  to  obscure  the  real  point--that  there  are  errors 
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in  the  data  base  which  have  an  adverse  impact  on  user 
satisfaction  with  MIS-produced  reports. 

The  Foundation  also  suggested  that  the  use  of  the  Pro¬ 
posal  Aging  Report  by  management  as  an  oversight  tool  over 
program  managers  may  have  caused  the  managers  to  resent  the 
aging  report,  and  therefore  judge  the  report  more  harshly 
in  our  user  survey.  However,  we  did  not  obtain  program  man¬ 
agers'  views  on  their  satisfaction  with  the  Proposal  Aging 
Report — program  managers  were  not  asked  to  judge  this  report. 
We  asked  only  division  directors — the  primary  users  of  the 
aging  report--how  satisfied  they  were  with  the  report.  As 
a  result,  our  questionnaire  could  not  have  measured  the  re¬ 
sentment  program  managers  might  have  regarding  the  aging 
report  since  they  were  not  asked. 

Foundation  suggestion  for 
GAO  work  scope  qualification 

The  Foundation  suggested  that  we  include  in  the  letter 
transmitting  our  report  to  the  Congress  a  notation  regarding 
the  limitations  that  were  placed  on  our  work.  We  had  already 
included  a  paragraph  qualifying  our  work  scope  in  the  intro¬ 
ductory  section  of  the  report  (see  paragraph  1  on  page  2  of 
appendix  I).  We  have  also  included  this  work  scope  qualifi¬ 
cation  in  the  introductory  section  of  the  basic  transmittal 
letter  to  the  chairman. 

Analysis  of  FEDSIM  report 
on  MIS  response  time 

The  Foundation  stated  that  the  FEDSIM  tests  show  system 
response  time  to  be  in  an  acceptable  range  and  that  the  tests, 
as  well  as  the  agency's  recent  experience,  do  not  support 
the  implication  in  the  GAO  report  that  only  20  terminals  can 
be  concurrently  active  on  the  system.  The  Foundation's  letter 
was  dated  June  15,  1979,  prior  to  FEDSIM's  final  report.  The 
final  report,  dated  September  1979,  acknowledges  the  occur¬ 
rence  of  slow  system  response  times  and  makes  many  recommenda¬ 
tions  for  improving  overall  system  performance.  The  study 
does  not  characterize  current  system  performance  as  being 
either  acceptable  or  unacceptable.  The  study  reports  that 
hardware  performance  was  generally  good  during  the  controlled 
testing  and  was  not  a  constraint  to  improved  system  perfor¬ 
mance,  but  that  better  management  of  available  computer 
resources  would  improve  system  performance. 

GAO's  reference  to  diminished  system  response  time  capa¬ 
bility  when  20  or  more  terminals  are  in  use  was  based  on 
information  received  by  interviewing  Foundation  personnel. 

Our  inquiry  showed  that  system  managers  did  not  know  the  cause 
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of  the  apparent  slow  response  time  problem  and  may  have,  in 
their  speculation,  attributed  the  problem  to  numerous  related 
or  unrelated  causes.  We  conclude  from  the  FEDSIM  study  that 
the  number  of  terminals  that  can  be  concurrently  active  and 
the  response  time  at  those  terminals  is  dependent  on  many 
variables,  including  the  number  of  users  and  the  difficulty 
of  their  procedures,  the  competition  between  data  entry  per¬ 
sonnel  and  other  users  for  computer  resources,  system  sched¬ 
uling  procedures,  and  others.  We  also  conclude  from  FEDSIM 
that  there  are  times  when  system  response  time  is  poor.  The 
study  indicates  that  under  controlled  testing  it  is  possible 
to  have  more  than  20  terminals  concurrently  active.  However, 
the  day  to  day  operations  of  the  Foundation  are  a  real  world 
environment  where  the  computer  system  must  respond  to  uncon¬ 
trolled  demands  which  we  believe  can  yield  situations  where 
only  20  terminals  could  be  concurrently  active.  Improved 
system  accountabil ity  data  would  be  needed  to  show  how  fre¬ 
quently  this  condition  actually  occurs. 

We  believe  it  is  important  to  note  the  FEDSIM  study's 
qualification  that  its  test  results  depended  strongly  on  the 
workload  and  operational  environment  measured  under  quali¬ 
fied  conditions.  We  observed  that  some  of  the  qualifica¬ 
tions  may  limit  the  study's  conclusions  with  respect  to  the 
Foundation's  actual  daily  operation.  For  example,  JASON, 
one  of  the  major  systems  tested,  lacked  the  software  instru¬ 
mentation  necessary  to  collect  terminal  response  times.  Also, 
the  test  methodology  used  in  the  system  loading  experiments 
excluded  batch  processing  from  the  workload  applied  to  the 
computer  system.  The  combined  effect  of  batch  processing 
and  normal  user  demands  on  limiting  the  number  of  terminals 
concurrently  active  is  unknown. 

In  its  discussion  of  available  computer  resources,  the 
FEDSIM  study  generally  assumes  24-hour  system  availability. 

If  all  the  Foundation's  procedures  could  be  distributed  over 
a  24-hour  period,  undoubtedly  system  overloads  could  be 
minimized.  But  in  practice,  the  Foundation  has  a  prime  day¬ 
time  peak  loading  period  which  is  a  period  of  great  user  de¬ 
mand  to  which  the  system  must  respond  successfully.  As  the 
study  notes,  the  central  processing  unit  resources  and  memory 
resources  are  almost  fully  utilized  during  this  day-time 
period.  > 

} 

We  also  observed  that  the  system  loading  experiments 
used  participants  and  representative  workloads  selected  by 
Foundation  personnel.  Such  an  approach  could  introduce  agency 
bias  into  the  testing  in  that  all  aspects  of  a  typical  work¬ 
load  environment  may  not  have  been  represented. 
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We  believe  that  the  FEDSIM  study  will  be  useful  to  the 
Foundation  in  strengthening  the  management  of  its  system.  Be¬ 
cause  of  the  limitations  of  the  study,  however,  we  believe  it 
would  not  be  surprising  to  find  system  users  experiencing  the 
same  response  time  problems  we  reported,  despite  FEDSIM's 
conclusions  that  response  time  is  generally  good  and  that 
system  hardware  is  adequate  to  handle  the  current  workload. 

The  difference  lies  in  assessing  system  components  under 
controlled  conditions  as  FEDSIM  did,  and  the  results  of  the 
system  operating  in  a  normal  work  environment. 

Regardless  of  the  cause,  users  will  react  negatively 
to  slow  response  time  at  the  terminals  even  if  they  are 
apprised  of  possible  causes.  The  Foundation's  task,  as  we 
see  it,  is  to  eliminate  or  minimize  what  Foundation  personnel 
perceive  as  a  system  response  time  problem  regardless  of  the 
exact  number  of  terminals  involved.  We  believe  FEDSIM's 
recommendations  should  be  helpful  in  doing  so.  (See  attach¬ 
ment  VI  for  excerpts  from  the  FEDSIM  report  showing  report 
abstract  and  FEDSIM's  conclusions  and  recommendations  re¬ 
garding  the  Foundation's  MIS.) 

Assessing  user  needs 

Although  the  Foundation  states  in  its  letter  that  it 
has  taken  steps  to  address  user  needs  by  establishing  an  ad¬ 
visory  group,  we  believe  the  Foundation  should  insure  that 
the  group's  actions  will  be  adequate  to  provide  for  user  needs 
assessment.  As  we  have  recommended,  a  performance  evaluation 
program  is  also  needed  in  addressing  user  needs.  The  impor¬ 
tance  of  conducting  user  needs  assessment  cannot  be  over¬ 
emphasized.  It  helps  insure  that  the  system  is  providing  the 
services  intended  and  that  the  services  are  accurate  and 
reliable. 

RECOMMENDATIONS 


We  recommend  that  the  Director  of  the  National  Science 
Foundation  improve  MIS  by  taking  actions  in  the  following 
areas : 

— System  Response  Time — Resolve  expeditiously  the  slow 
system  response  time  problem  which  is  frustrating  sys¬ 
tem  users,  and  reassess  the  need  for  acquiring  addi¬ 
tional  hardware. 

— Data  Quality  Control — Assess  MIS's  data  quality 
and  establish  procedures  for  periodically  testing 
and  maintaining  data  quality  to  insure  high  reli¬ 
ability. 
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— Long-Range  Planning — Develop  and  maintain  a  long-range 

system  and  hardware  planning  process  to  insure  orderly  - 

and  systematic  development  of  MIS.  The  planning 
process  shoul 3  include  top  management  participation; 
provide  a  reasonable  5-year  projection  of  system  re¬ 
quirements  and  costs,  which  would  include  life-cycle 
cost  analysis;  establish  quantified  goals  and  priori¬ 
ties  for  work  projects;  and  set  milestone  completion 
dates . 

— Performance  Evaluation — Develop  and  implement  perform¬ 
ance  evaluation  procedures  to  determine  and  control 
the  efficiency  of  the  system  in  meeting  its  goals. 

Such  procedures  should  assess  user  needs  and  insure 
system  cost  effectiveness. 

— Overall  System  Maintenance — Assess  MIS  short-  and  mid¬ 
range  management  efforts  to  insure  that  areas  such  as 
general  system  security,  system  documentation,  and 
training  receive  adequate  attention.  Also,  the  Director 
should  strengthen  the  internal  audit  staff  capability 
to  monitor  MIS  performance. 
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NATIONAL  SCIENCE  FOUNDATION 
WASHINGTON.  DC  20550 


nsf 

June  15,  1979 

OFFICE  OF  THE 
DIRECTOR 


Mr.  Harry  S.  Havens 
Director 

Program  Analysis  Division 

U.  S.  Government  Accounting  Office 

Washington,  D.C.  20548 

Dear  Mr.  Havens: 

I  appreciate  the  opportunity  to  comment  on  your  office's 
proposed  report  on  the  National  Science  Foundation's 
Management  Information  System  (MIS) .  We  appreciate  the 
importance  of  a  responsive,  accurate  MIS  and  are  deter¬ 
mined  to  continue  improving  our  system's  effectiveness 
for  both  the  NSF  staff  and  the  Congress.  Your  report 
will  be  useful  to  us  as  we  develop  plans  for  future 
improvements  in  this  important  resource. 

However,  I  am  concerned  that  the  report's  usefulness  may 
be  diminished  by  the  audit  team's  frequent  reliance  on 
anecdotal  evidence  rather  than  empirical  data.  Interviews 
and  questionnaires  help  assess  user  satisfaction,  of  course, 
but  they  also  can  make  it  difficult  for  an  audit  team  to 
distinguish  between  legitimate  system  problems  and  other 
factors  that  influence  user  satisfaction.  An  example  of 
this  effect  is  the  user  survey  conducted  by  your  audit 
team.  The  survey  revealed  that  82%  of  the  respondents 
are  "satisfied"  with  the  accuracy  and  reliability  of  the 
"Workload  Status  Report,"  while  only  29%  are  "satisfied" 
with  the  "Proposal  Aging  Report."  In  fact,  both  reports 
are  derived  simultaneously  from  the  same  integrated  data 
base.  Thus,  the  differences  in  user  perception  for  these 
two  reports  cannot  be  attributed  solely  to  the  quality  of 
our  integrated  data  base.  The  "Workload  Status  Report" 
contains  information  on  all  proposals  received  by  the 
Foundation  during  the  current  fiscal  year.  The  "Proposal 
Aging  Report"  contains  proposals  received,  less  those  on 
which  an  action  has  been  entered  correctly  in  the  MIS. 

Thus,  overall  data  base  errors  tend  to  be  concentrated  in 
(  this  report  simply  as  a  result  of  th#  !WLuie  of  Um  report. 

(See  GAO  response  in  Appendix  I,  pp.  24,  25.) 
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These  same  proposals  and  errors  also  are  present  in  the 
"Workload  Status  Report"  but,  because  of  the  larger  volume 
of  information  present,  they  are  not  as  easily  recognized. 
Moreover,  the  aging  report  is  used  by  top  management  to 
ensure  the  prompt  processing  of  all  proposals  submitted 
to  the  Foundation.  The  report  serves  as  a  valuable  tool 
for  identifying  "old"  proposals  and  bottlenecks  in  the 
proposal  processing  stream.  This  use  of  the  report  as 
a  management  oversight  tool  may  be  resented  on  occasion 
as  an  intrusion  into  the  responsibility  of  program 
managers,  and  therefore  may  be  judged  more  harshly  in 
user  attitude  surveys  than  less  sensitive  reports. 

(See  GAO  response  in  Appendix  I,  pp.  24,  25.) 

I  recognize  that  the  audit  team  did  not  have  time  for  an 
exhaustive  study  of  the  NSF  system  and  had  to  rely  on  less 
time-consuming  techniques  for  data  gathering.  I  do  believe, 
however,  that  this  limitation  should  be  acknowledged  explic¬ 
itly  in  the  letter  transmitting  the  report  to  Congress  and 
in  the  conclusion  of  the  repos  (See  GAO  note  below.) 

I  also  would  like  to  comment  briefly  on  the  report's 
specific  recommendations. 

The  managers  of  the  NSF  system  are  aware  of  the  importance 
of  system  response  time  as  a  measure  of  system  reliability. 
The  users  of  our  MIS,  as  noted  in  the  report,  would  like 
a  response  time  of  under  5  seconds,  and  our  system  manage¬ 
ment  agrees  that  this  is  a  desirable  goal.  During  the 
past  three  years,  the  Federal  Simulation  Center  (FEDSIM) 
has  conducted  two  computer  performance  evaluations  at  NSF, 
the  latest  commencing  in  January,  1979.  A  major  portion 
of  the  most  recent  study  was  designed  to  assess  the  inter¬ 
action  between  our  MIS  executive  and  the  timesharing  sub¬ 
system.  Controlled  tests  consisting  of  adding  sequentially 
up  to  75  user  terminals  were  conducted  several  times  to 
assess  the  impact  on  the  system,  including  response  time. 
Response  time  during  these  tests  was  considered  by  FEDSIM 
to  be  in  an  acceptable  range.  These  quantitative  tests, 
as  well  as  our  own  recent  experience ,  do  not  support  the 
implication  in  the  GAO  report  that  only  20  terminals  can 
be  concurrently  active  on  the  NSF  system.  The  final 
FEDSIM  report  will  be  published  in  June,  1979  and  will 
provide  detailed  data  on  system  response  capabilities. 

(See  GAO  response  in  Appendix  I,  pp.  25,  26,  and  27.) 

GAO  Note:  Qualification  statement  included  on  page  1  of  the 
letter  to  the  Chairman.  Our  conclusions  are  based 
on  our  audit  work  which  is  qualified. 
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We  intend  in  the  future  to  conduct  computer  performance 
evaluations  on  a  regular  basis  so  that  we  maintain  an 
optimized  system. 

Data  quality  is  of  continual  concern  to  us,  since  our 
MIS  is  founded  on  the  premise  of  supplying  accurate  and 
timely  information  to  all  levels  of  users.  We  will 
initiate  a  data  quality  assessment  program  in  the  near 
future.  This  program  will  identify  problem  areas,  and 
also  may  involve  adding  controls  to  individual  application 
software  as  warranted. 

We  recognize  long  range  planning  as  an  area  needing  more 
attention  in  the  future.  We  also  recognize  the  need  for 
close  coordination  of  ADP  plans  with  overall  NSF  plans. 
Indeed,  in  order  to  determine  user's  needs,  an  advisory 
group  was  established  in  January,  1979  to  assist  in 
bringing  user's  requirements  to  the  attention  of  MIS 
management.  This  advisory  group  has  recommended  the 
creation  of  a  planning  team  comprised  of  computer  spec¬ 
ialists,  user  representatives  and  management.  This 
recommendation  is  being  adopted,  and  I  expect  it  to  lead 
to  greatly  improved  planning,  both  for  the  next  fiscal 
year  as  well  as  the  longer  term. 

Performance  evaluation  is  a  difficult  art.  It  has  rarely 
been  successfully  implemented  in  federal  agencies.  Never¬ 
theless,  we  recognize  the  need  to  evaluate  system  perform¬ 
ance  and  assess  the  costs  of  meeting  user  needs.  Thus, 
beginning  in  July,  1979  we  will  begin  experimenting  with  a 
new  project  management  software  system.  This  software  tool 
should  help  us  collect  data  and  produce  reports  on  the 
costs  of  data  processing  and  use  of  our  resources.  In 
addition,  we  plan  to  implement  a  computer  accounting  system. 
Thus,  in  the  future,  cost  data  should  be  available  for 
software  development,  maintenance,  and  computer  time. 

Computer  security  is  an  important  issue,  and  we  have 
submitted  our  security  plans  in  response  to  OMB  directives. 
Moreover,  in  response  to  a  previous  GAO  report,  we  recently 
appointed  a  computer  security  administrator  in  the  office 
of  the  Assistant  Director  for  Administration.  The 
administrator  is  responsible  for  developing  a  security 
plan  that  will  involve  users,  computer  technicians,  and 
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internal  audit  staff.  In  addition,  the  NSF  Audit  Office 
recently  hired  a  computer  specialist  who  will  work  on  ADP 
audits.  A  software  certification  program  also  has  been 
established  to  improve  the  quality  of  computer  products. 
This  program  will  define  technical  documentation  standards, 
and  then  analyze  all  software  products  to  ensure  compliance 
with  these  standards . 

Again,  I  wish  to  express  my  appreciation  for  the  efforts 
of  GAO  and  to  assure  you  that  they  will  assist  us  in 
improving  the  Foundation's  computer  system.  I  look  forward 
to  seeing  the  final  report. 

Sincerely  yours. 


Richard  C.  Atkinson 
Director 
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PREFACE 


This  report  is  based  on  the  analysis  of  a  large  volume 
of  data.  The  results  address  the  measurement  of  the  National 
Science  Foundation's  Honeywell  6060  computer  system. 

Because  the  results  depend  strongly  on  the  workload  and 
operational  environment  measured,  generalizing  the  recom¬ 
mendations  beyond  the  system  described  or  extracting  con¬ 
clusions  without  their  respective  qualifying  conditions  is 
not  possible.  Questions  related  to  this  report  or  to  the 
possibility  of  extending  the  stated  conclusions  or  recom¬ 
mendations  should  be  addressed  to  the  study's  author  at  the 
Federal  Computer  Performance  Evaluation  and  Simulation 
Center  (FEDSIM). 


41 


ATTACHMENT  VI 


ATTACHMENT  VI 


ABSTRACT 


From  12  February  to  13  April  1979,  the  Federal  Computer 
Performance  Evaluation  and  Simulation  Center  (FEDSIM) 
conducted  a  computer  performance  evaluation  study  of  the 
Honeywell  6060  computer  system  operated  by  the  National 
Science  Foundation  (NSF).  This  project  report  presents  the 
objectives,  methodology,  findings,  conclusions,  and  recom¬ 
mendations  of  the  study.  This  Abstract  contains  the  most 
important  conclusions  and  recommendations  based  on  NSF's  (1) 
workload  characteristics,  (2)  system  response,  (3)  system 
design,  (4)  system  utilization  levels,  and  (5)  operational 
procedures. 

WORKLOAD  CHARACTERISTICS 

(1)  NSF  should  establish  formal  response  and  turnaround 
time  requirements  for  system  users. 

(2)  Much  batch  workload  is  processed  during  the  day 
shift  when  on-line  systems  are  also  demanding  system 
resources.  NSF  6hould  evaluate  the  requirement  for 
most  batch  activities  to  process  during  the  day 
shift. 

(3)  Programmers  should  make  more  use  of  system  loadable 
files  and  libraries  to  reduce  the  overhead  required 
to  reload  programs  each  time  they  are  used. 

(4)  The  use  of  BASIC  programs  on  the  TSS  system  should 
be  re-evaluated  to  see  if  these  programs  could  be 
more  efficiently  coded  into  loadable  TSS  subsystems. 

SYSTEM  RESPONSE 

(1)  TSS  response  times  are  good.  Longer  response  times 
experienced  by  TSS  users  are  a  result  of  non-trivial 
requests  to  TSS  and  problems  (disk  errors,  etc.) 
that  stop  TSS  from  responding  to  all  users. 

(2)  JASON  response  times  are  more  a  function  of  JASON 
user  activity  (or  the  interaction  of  JASON  sub¬ 
systems  with  the  database)  that  a  function  of 
hardware  resource  availability. 
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SYSTEM  DESIGN 

(1)  NSF  should  re-evaluate  the  design  of  the  Office 
Management  System  to  more  closely  meet  the  needs  of 
system  users  and  to  reduce  the  system's  load  on  TSS. 

(2)  NSF  should  remove  JASON  data  entry  functions  from 
competition  with  other  JASON  users  to  provide  data 
entry  personnel  with  more  consistent  response. 

(3)  JASON  should  be  instrumented  to  provide  the  necessary 
information  to  better  evaluate  JASON  response  time 
and  the  interaction  between  user-written  JASON 
subsystems . 

SYSTEM  UTILIZATION  LEVELS 

(1)  Few  additional  CPU  resources  exist  during  the  day 
shift.  However,  CPU  resources  are  not  considered  to 
be  a  constraint  to  system  performance  because  (1) 
excess  CPU  resources  exist  during  non-prime  processing 
periods  which  may  be  used  by  jobs  now  executing 
during  the  day  shift  and  (2)  response  times  are 
generally  good  or  if  poor,  they  are  not  a  direct 
function  of  CPU  availability. 

(2)  Memory  is  almost  fully  used  during  the  day  shift 
causing  some  jobs  to  move  in  ami  out  of  memory 
repeatedly.  However,  memory  is  not  considered  to  be 
an  overall  constraint  to  system  performance  because 
excess  memory  resources  exist  during  non-prime 
processing  periods. 

(3)  Disk  and  tape  channel  utilization  are  not  a  constraint 
to  improved  system  performance. 

(4)  The  Datanet  355  front  processor  and  memory  utilization 
are  not  constraints  to  improved  system  performance. 

OPERATIONAL  PROCEDURES 


(1)  NSF  should  improve  the  collection  and  retention  of 

system  accounting  data.  Improved  reporting  procedures 
using  these  data  should  be  implemented  to  provide 
NSF  management  with  information  to  better  manage 
computer  resources. 
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(2)  NSF  should  examine  disk  configuration,  use  of  NOFMS 
disk  options,  and  system  file  placement  to  improve 
disk  efficiency. 

(3)  BASIC  users  should  use  the  "RON  »  filename"  option 
whenever  possible  to  reduce  the  resources  required 
by  the  BASIC  compiler. 

(4)  NSF  should  improve  communications  with  system  users 
to  (1)  notify  them  of  system  malfunctions,  etc.  and 
(2)  provide  guidance  to  improve  awareness  of  how  a 
user's  activity  impacts  response  times  and  system 
resource  use. 
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E.  SUMMARY  OF  CONCLUSIONS 


The  following  summary  of  conclusions  restate  conclusions 
presented  throughout  the  Findings  (Section  IV)  and  contain 
information  based  on  the  methodology,  findings,  and  con¬ 
clusions  of  the  system  loading  experiments.  FEDSIM  suggests 
that  Appendix  D  be  read  before  proceeding  with  this  Section. 
Each  conclusion  is  a  result  of  the  analysis  of  NSF's  (1) 
operational  procedures,  (2)  workload  characteristics,  or  (3) 
system  component  utilization  levels. 

1 .  More  accurate  accounting  data  and  improved  methods  for 
their  collection  and  retention  vrould  benefit  management  and 
evaluation  of  the  NSF  system. 

2.  Although  system  scheduling  procedures  were  good,  some 
factors  and  procedures  in  the  current  scheduling  design 
could  constrain  system  performance. 

3.  System  availability  was  good.  The  TSS  system  was 
available  for  most  of  its  scheduled  periods.  However,  the 
JASON  on-line  system  was  unavailable  to  users  a  greater 
percentage  of  the  time  which  intensified  JASON  workload  when 
the  JASON  system  was  available. 

4.  More  detailed  information  should  be  collected  and 
reported  concerning  system,  subsystem,  and  component  avail¬ 
ability  and  the  reason  for  periods  of  unavailability. 

5.  The  Office  Automation  System  does  not  meet  the  needs  of 
all  divisions  that  use  it.  Centralized  planning  and  control 
for  this  system  does  not  exist,  resulting  in  operational 
problems  and  potential  constraints  to  future  system  per¬ 
formance  . 

6.  Tape  management  procedures  were  generally  adequate,  but 
certain  occurrences  of  data  losses  due  to  the  inadvertent 
scratching  of  tapes  indicated  that  some  small  changes  are 
needed  in  this  area. 

7.  Most  batch  activities  are  executed  during  day  shift 
processing  periods  when  resource  consumption  of  on-line 
systems  is  greatest. 

8.  Most  activities  executed  require  small  amounts  of 
processor,  memory,  and  I/O  resources. 
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9.  Activities  requiring  two  tapes  imposed  a  significantly 
higher  load  on  the  tape  subsystem  than  did  activities 
requesting  other  amounts  of  tape  drive  resources. 

10.  The  number  of  aborted  activities  is  not  a  major  problem 
on  the  H-6060  system.  However,  almost  6%  of  processing 
resources  and  over  10%  of  memory  resources  are  lost  to 
aborted  activities. 

11.  Most  NSF  users  load  (link-edit)  production  programs  each 
time  they  are  executed,  instead  of  using  load  modules  (H*  or 
**  files)  and  production  libraries. 

12.  TSS  response  loads  were  moderate.  Most  NSF  users 
executed  TSS  subsystems  that  were  fewer  than  2 IK  words  in 
size . 

13.  The  largest  amount  of  TSS  processor  use  was  consumed  by 
users  of  the  BASIC  subsystem.  Users  compiled  programs  each 
time  they  were  used  instead  of  using  more  efficient  processing 
techniques. 

14.  Text  editor  functions  of  the  TSS  system  are  largely  used 
for  document  processing.  The  Text  Editor  subsystem  and  the 
RUNY  subsystem  (under  which  Office  Automation  programs  were 
run)  accounted  for  almost  45%  of  all  characters  transmitted 
by  the  TSS  system. 

15.  Peak  demand  for  system  resources  by  the  JASON  on-line 
system  and  the  TSS  system  occur  concurrently. 

16.  System  resources  are  not  a  constraint  to  additional 
workload  during  evening,  night,  and  weekend  processing 
periods. 

17.  CPU  resources  are  almost  fully  used  during  day  shift 
processing  periods.  These  levels  do  not  pose  a  significant 
constraint  to  response*  levels  of  on-line  systems  at  current 
user  activity  levels. 

18.  Memory  resources  are  almost  fully  used  during  day  shift 
processing  periods.  This  level  of  utilization  is  not  a 
constraint  to  system  throughput  for  most  batch  jobs.  However, 
some  jobs  exhibit  significant  swap  activity  which  adds  to  job 
elongation  and  system  overhead  use  of  processing  and  disk 
resources . 

19.  Disk  channel  availability  is  not  a  constraint  to  system 
performance . 
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20.  Disk  controller  utilization  is  not  a  constraint  to 
system  performance. 

21.  System  file  placement  was  not  a  major  constraint  to 
system  performance.  However,  some  small  changes  in  the  way 
system  files  are  allocated  to  disk  devices  could  yield  some 
performance  improvements. 

22.  Seek  activity  on  some  JASON  disk  packs  may  vary  enough 
to  degrade  response  times  of  the  JASON  system. 

23.  System  overhead  can  be  reduced  by  placing  system  modules 
.MFS03  and  .MFS09  permanently  in  memory  and  implementing  SSA 
Cache  Memory  to  reduce  system  module  loads  from  disk. 

24.  Tape  channel  configuration  is  not  a  constraint  to  system 
performance . 

25.  The  present  tape  controller  is  not  a  constraint  to 
system  performance. 

26.  The  number  of  tape  drives  configured  on  the  system  is 
generally  adequate  for  processing  the  current  NSF  workload. 
However,  a  constant  demand  exists  for  additional  tape 
resources.  This  demand  may  be  reduced  somewhat  by  changes 
in  memory  swap  rates  and  job  control  language  procedures  to 
reduce  the  number  of  tapes  allocated  but  idle. 

27.  The  Datanet-355  front-end  processor  is  not  a  constraint 
to  on-line  response  times. 

28.  JASON  on-line  response  times,  at  current  activity 
levels,  are  more  a  function  of  the  interaction  of  JASON 
procedures  than  hardware  resource  availability. 

29.  TSS  response  times  are  good.  Comments  about  slow  TSS 
response  seem  to  be  more  a  factor  of  the  type  of  function 
the  user  is  doing  than  overall  TSS  performance.  Occasions 
when  system  processing  is  interrupted  (disk  errors,  accounting 
file  problems,  line  problems,  etc.)  are  being  interpreted  by 
users  as  poor  TSS  or  JASON  response  because  these  users  are 
not  made  aware  of  the  occurrance  of  system  problems. 
Conversely,  users  experiencing  problems  are  not  always 
making  computer  operations  personnel  aware  of  these  problems. 
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V.  RECOMMENDATIONS 


A.  OPERATIONAL  PROCEDURES 

FEDSIM  recommends  that  the  policies  and  procedures  for 
system  accounting  be  enhanced  to  provide  NSF  personnel  with 
better  information  with  which  to  manage  computer  and  man¬ 
power  resources.  FEDSIM  identified  three  main  areas  where 
system  accounting  could  be  improved: 

a.  Accounting  for  User  Activity.  FEDSIM  recommends 
that  the  $IDENT  card  format  be  modified  and  expanded  to 
include  detailed  information  on  the  submitting  agency, 
project  code,  test  or  production  status,  and  type  of 
system  (JASON,  TSS,  or  normal  batch,  etc).  Figure  V-l 
outlines  a  sample  $IDENT  card  format. 


$  IDENT  AAAAAABBBB/ CD/EEEEEE/ , FFFFFFFFF ,GGG . . . 


AAAAAA  *  Office  Identification 

BBBB  *  Employee  Number 

C  *  System  Code 

J* JASON 
L=TSS 

B=BATCH,  etc. 

D  *  Production  Code 

P=Production 
T=Test 

S=Special  (one  time  runs,  etc.) 
EEEEEE  *  Program  Identification  (JASON  MODULE, 
FFFFFFFFF  *  Batch  Distribution  Banner 
GGG...  *  Free  Field  for  Remainder  of  Card 


etc. ) 


SAMPLE  $  IDENT  CARD  FORMAT 
FIGURE  V-l 

FEDSIM  also  recommends  that  the  "ACCOUNT?"  question 
be  implemented  for  TSS  users  and  Fields  A  through  E  of 
Figure  V-l  be  required  of  all  workload  submitted  under 
TSS.  The  .MGNAT  module  (system  input  validity  check), 
the  .MSCAN  (user  scheduling),  and  .MSCN1  (control 
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card  examination)  modules  should  be  used  in  conjunction 
with  each  other  to  cross-reference  the  office  identi¬ 
fication,  program  identification,  and  user  identification 
of  each  batch  program  and  TSS  user  to  ensure  valid  data 
are  being  collected  by  the  accounting  system.  This 
cross-reference  check  with  the  user  identification 
(which  is  used  in  conjunction  with  a  password)  will 
provide  security  for  codes  used  on  the  $ I DENT  card  yet 
will  not  require  important  data  on  the  $IDENT  card  to  be 
overlayed  and  lost  for  later  accounting.  FEDSIM  recommends 
that,  once  enhanced  system  accounting  has  been  implemented, 
users  and  their  management  be  made  aware  of  their  own 
activity  levels  and  resource  consumption  on  a  monthly 
basis.  The  knowledge  gained  from  monthly  reports 
provides  management  with  the  ability  to  track  workload 
growth,  aid  capacity  planning,  identify  candidates  for 
program  optimization,  and  "police"  the  use  of  accounting 
codes  and  parameters. 

b.  Accounting  for  System  Availability.  FEDSIM  recommends 
that  the  current  method  of  tracking  system,  TSS,  and 
JASON  availability  be  expanded  to  include  specific  times 
when  outages  are  experienced,  to  account  for  component 
availability  (front-end  processors,  etc.),  and  to  trace 
the  causes  for  these  outages.  FEDSIM  recommends  that 
NSF  personnel  design  and  implement  a  form  (preferably 
with  sequence  numbers  for  accountability)  for  computer 
operations  personnel  to  use  when  problems  arise.  This 
form  will  provide  a  record  of  system  problems  and  serve 
as  a  tracer  for  subsequent  disposition  of  problems. 

Figure  V-2  is  a  sample  of  the  type  of  form  recommended. 

Expanded  accountability  will  ensure  that  each 
problem  is  noted  and  will  allow  NSF  management  to  commit 
personnel  and  computer  resources  more  effectively  to 
solve  those  problems  identified  as  causing  significant 
inconvenience  to  users. 

FEDSIM  also  recommends  that  NSF  personnel  closely 
evaluate  data  provided  by  test  and  diagnostic  software 
to  trace  problems  on  tape  and  disk  devices  and  monitor 
if  and  when  these  devices  are  becoming  significant 
constraints  to  system  performance. 

FEDSIM  also  recommends  that  information  concerning 
system  problems  be  made  available  to  users  so  that  they 
may  possibly  relate  their  experiences  (bad  response,  bad 
turnaround,  etc.)  to  system  malfunctions. 
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DATE  _  TIME  UP  _ 

OPERATOR  _ _ 

MAJOR  PROBLEM  AREA 

H-6060  £J  D-355 

rj  DISK  _  rj  "APE 

rj  JASON  rj  TSS 

rj  OTHER  _ 

DISK  PACK  OR  TAPE  REEL  #  _ 

SYMPTOMS:  _ 


TIME  DOWN  _ 

LOG  NUMBER 


rj  MCS-30 

n  memory 

rj  ACCOUNTING 


DUMP?  n  YES  jTJ  NO  TAPE  # 

PRINTED?  £J  YES  £J  NO 

ANALYSIS:  _ _ 


DISPOSITION: 


SAMPLE  SYSTEM  PROBLEM  TRACKING  FORM 
FIGURE  V-2 
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c.  Collection  and  Retention  of  Accounting  Data.  FEDSIM 
recommends  that  NSF  personnel  implement  a  procedure  to 
ensure  that  accounting  data  are  properly  collected  and 
maintained.  Table  V-l  shows  the  accounting  record  types 
normally  produced  by  GCOS.  Figure  V-3  presents  a 
sample  accounting  merge  procedure  designed  to  collect 
and  retain  accounting  data.  This  procedure  will  retain 
certain  record  types  only  as  long  as  necessary  to  meet 
the  information  needs  of  NSF  computer  operations.  For 
example,  accounting  records  such  as  Type  6  (system  dump 
records)  are  important  for  the  analysis  of  TSS  failures, 
etc.,  but  are  not  needed  for  workload  purposes.  These 


RECORD  TOTE 


RECORD  DESCRIPTION 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 
19 

20-100 

101-OVER 


Activity  Accounting  Record 

SYS  OUT  Report  Writer  Accounting  Record 

I/O  Error  Accounting  Record 

TQLTS  Statistical  Record 

System  Input  Accounting  Record 

Program  Dump  or  Operator  SNAP  Accounting  Record 

Time  Sharing  System  Accounting  Record 

Activity  Accounting  Record  Overflow 

Restart  Accounting  Record 

Transaction  Processing  System  Statistical  Record 
Reserved 

Operator-initiated  Record 
Reserved 

Time  Sharing  Resource  Data  Record 
File  Managanent  Supervisor  Accounting  Record 
FMS  Master  Function  Audit  Record 
FMS  User  Function  Audit  Record 

FMS  Allocate/Deallocate  of  Secured  File  Audit  Record 
Time  Sharing  Test  and  Measurement  Record 
Reserved  for  CCOS 
User-Defined  Record  Types 


GCOS  ACCOUNTING  RECORDS 
TABLE  V-l 
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records  are  not  merged  into  the  accounting  data  but  are 
processed  at  the  same  time  the  daily  merge  is  done. 

Data  such  as  FMS  audit  records  (Types  16,  17,  and  18) 
are  merged  into  daily  accounting  tapes  so  that  questions 
concerning  file  security  and  access  may  be  answered; 
yet,  these  records  are  deleted  by  the  weekly  accounting 
merge  when  they  are  no  longer  needed.  Retention  cycles 
outlined  in  Figure  V-3  are  designed  to  provide  adequate 
backup  capability  and  ensure  that  data  are  available  to 
answer  ad  hoc  inquiries  for  a  reasonable  time  after  data 
have  been  merged  into  more  generalized  formats.  The 
weekly  and  monthly  accounting  merge  tapes  contain  only 
those  data  needed  for  user  accounting  and  capacity 
management  functions,  thereby  condensing  the  volume  of 
data  on  these  tapes.  FEDSIM  has  provided  NSF  personnel 
with  an  accounting  merge/extract  program  that  may  be 
tailored  to  accomplish  this  recommendation. 

FEDSIM  recommends  that  the  current  system  scheduling 
philosophy  on  the  NSF  computer  system  be  altered  to  reflect 
priorities  in  batch  workload  and  to  eliminate  possible 
performance  problems. 

a.  Establishment  of  Operations  Class.  FEDSIM  recom¬ 
mends  that  NSF  establish  an  operations  scheduler  queue 
for  all  jobs  spawned  by  computer  operations  personnel 
(VIDEO,  HEALS,  etc.).  This  class  will  provide  a  separate 
pathway  into  the  system  for  operations  jobs,  thus 
eliminating  them  from  competing  with  normal  batch 
workload  in  the  system  scheduler  queues.  Also,  the 
operations  queue  will  ensure  that  jobs  required  by  the 
operator  can  become  candidates  for  execution  quickly. 

Disk  accounting  purge  programs  should  also  be  scheduled 
through  this  queue  to  ensure  their  prompt  execution. 

b.  Dse  of  JORDER,  JCLASS.  FEDSIM  recommends  that 
system  operators  alter  their  procedures  for  extracting 
jobs  from  scheduler  queues.  The  computer  operator  often 
used  the  console  verb  JSCHED  to  manually  start  workload 
from  other  scheduler  queues.  Jobs  that  required  signif¬ 
icant  amounts  of  resources  were  rescheduled  into  the 
express  queue  and  occupied  program  numbers  normally 
reserved  for  short,  quick  jobs.  FEDSIM  recommends  that 
computer  operators  use  the  JCLASS  console  verb  to  raise 
and  lower  the  number  of  jobs  allowed  from  each  class. 

This  procedure  will  ensure  that  jobs  will  run  from  the 
class  to  which  they  were  assigned.  If  it  is  necessary 
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to  run  a  particular  job  first,  the  computer  operator 
should  use  the  JORDER  verb  to  reorder  the  system  scheduler 
file  class  before  allowing  a  job  to  execute  from  that 
class . 

c.  Design  of  System  Scheduler  Classes.  FEDSIM  recom¬ 
mends  that  NSF  personnel  modify  the  system  scheduling 
philosophy  to  more  closely  reflect  workload  character¬ 
istics  as  outlined  in  the  findings  section  (Section  IV) 
of  this  report.  Those  findings  have  shown  that  most  NSF 
batch  activities  were  of  short  duration  and  required 
small  amounts  of  memory.  Additionally,  jobs  requesting 
one  tape  made  little  demand  on  tape  channel  resources, 
whereas  jobs  requesting  two  tape  drives  showed  signif¬ 
icantly  more  tape  channel  activity.  FEDSIM  recommends 
that  a  differentiation  be  made  between  jobs  that  request 
one  tape  drive  and  those  that  request  more  tape  resources. 

FEDSIM  recommends  that  the  current  configuration  of  the 
H-6060  be  modifed  slightly  to  balance  MFC  utilization  and  to 
reduce  the  chance  of  disk  errors  incapacitating  the  entire 
disk  subsystem.  Currently,  all  26  disk  drives  are  defined 
on  Channel  8  of  the  IOM  with  Channels  10,  12,  and  14  acting 
as  alternate  paths  to  the  disk  drives.  Although  this 
configuration  is  not  a  significant  performance  contraint, 

GCOS  builds  one  linked  list  for  all  I/O  activity  to  disk 
devices.  When  disk  errors  occur,  they  are  linked  to  the 
front  of  this  list  and  constrain  I/O  activity  for  all  other 
programs  and  disk  devices.  Figure  V-4  shows  a  suggested 
configuration  to  reduce  this  contention.  Since  most  disk 
activity  is  evenly  balanced,  NSF  personnel  should  determine 
what  devices  should  be  on  each  disk  string.  One  possiblity 
would  be  to  segregate  TSS  swap  files  from  JASON  removable 
packs  to  reduce  the  possibility  that  these  two  on-line 
systems  could  conflict  with  each  other  if  disk  errors 
occurred.  The  configuration  change  outlined  in  Figure  V-4 
is  a  logical  change  to  GCOS  and  does  not  require  hardware 
modifications . 

FEDSIM  recommends  that  the  use  of  the  NOFMS  option  be 
re-evaluated  at  NSF.  Disk  packs  DP5  and  DP8  were  defined  to 
not  contain  permanent  files.  Interviews  with  NSF  personnel 
indicated  that  this  NOFMS  option  was  used  to  identify  more 
temporary  file  space  for  system  users.  GCOS  system  loading 
algorithims  do  not  take  NOFMS  options  into  consideration 
when  spreading  files  among  disk  devices.  The  NOFMS  option 
may  restrict  the  number  of  disk  packs  available  to  permanent 


ATTACHMENT  VI 


ATTACHMENT  VI 


CONFIGURATION 

$  IOM-O  PUB-08,  DISKn91,UNITS-13... 

$  IOM-O  PUB-1 2 , D1SK*I 91 , UNITS- 1 3 . . . 

$  XBAR  IOM-O, PUB-8, PUB-14, PUB-9, PUB-15 

$  XBAR  IOM-O, PUB1 2, PUB-10,PUB1 3, PUB-11 


SAMPLE  DISK  CONFIGURATION 


FIGURE  V- 4 
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disk  files,  thus  increasing  permanent  file  contention,  but 
without  guaranteeing  that  space  will  be  effectively  used  on 
the  NOFMS  packs.  NSF  personnel  should  evaluate  whether  a 
significant  number  of  jobs  require  the  large  contiguous  disk 
areas  provided  by  NOFMS  packs  (for  sorting,  etc.).  If  large 
temporary  disk  storage  areas  are  needed  infrequently,  NSF 
personnel  should  consider  using  removable  packs  for  these 
jobs.  If  these  areas  are  not  needed,  the  NOFMS  options 
should  be  removed  to  further  balance  permanent  and  temporary 
file  placement. 

FEDSIM  recommends  that  NSF  personnel  periodically  check 
to  ensure  that  four-way  interlacing  is  being  used  for  memory 
accesses  on  the  H-6060  computer  system.  FEDSIM  noted  that 
after  new  hardware  had  been  installed  on  the  system,  inter¬ 
lacing  had  been  turned  off.  FEDSIM  has  found  in  various 
computer  studies,  including  the  previous  study  at  NSF,  that 
four-way  interlacing  makes  a  significant  improvement  in  the 
CPU's  capability  to  quickly  access  memory  locations. 

FEDSIM  recommends  that  the  placement  of  TSS  program  and 
swap  files  be  examined  with  each  software  release  to  ensure 
that  these  files  reside  on  separate  disk  devices. 

FEDSIM  noted  that  NSF,  because  of  its  relationships  with 
educational  institutions,  has  a  rapid  turnover  of  personnel. 
FEDSIM  recommends  that  seminars  concerning  the  proper  use  of 
the  system  be  given  to  new  personnel  to  ensure  that  they  are 
cognizant  of  how  their  activity  will  affect  system  resource 
consumption  and  availability.  Instruction  should  include 
operational  procedures,  proper  use  of  JASON  and/or  TSS, 
reasonable  programming  standards,  and  where  to  seek  assistance 
for  programming  and/or  procedural  problems. 

FEDSIM  noted  that  NSF  operational  procedures  require  a 
SAVE  SINCE  of  the  permanent  file  system  to  be  run  three 
times  a  day.  This  save  processing  copies  to  tape  all  files 
that  have  been  created  or  altered  since  the  last  SAVE  SINCE. 
FEDSIM  regarded  this  procedure  as  a  possible  constraint  to 
on-line  performance  since,  during  SAVE  processing,  the  File 
Management  Supervisor  (FMS )  hash  bits  are  closed  for  the 
user  master  catalog  being  saved.  With  only  31  FMS  bits, 
many  different  USERIDS  may  use  the  same  hash  bit.  When 
these  bits  are  "closed,"  the  ability  for  users  to  sign-on  to 
TSS  or  access  files  is  delayed.  FEDSIM  recommends  that  NSF 
personnel  track  how  often  a  SAVE  SINCE  is  actually  used  to 
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restore  files  and  consider  SAVE  SINCE  processing  once  a  day 
after  prime  shift  processing  periods.  If  additional  SAVE 
SINCE  processing  is  necessary,  NSF  personnel  should  evaluate 
the  need  to  save  user  files  on  an  individual  user  basis. 

FEDSIM  recommends  that  tape  management  procedures  be 
modified  to  reduce  the  possibility  of  inadvertent  misuse  of 
tape  reels.  FEDSIM  noted  that  some  tape  reels  were  accidentally 
scratched  before  their  release  date,  although  this  is  not  a 
major  problem.  This  problem  may  be  reduced  by  expressing 
the  retention  period  on  the  external  tape  label  as  a  date 
rather  than  the  number  of  days  in  the  retention  cycle.  This 
change  would  reduce  the  possibility  of  mathematical  error 
when  deciding  whether  or  not  a  tape  volume  should  be  scratched. 
As  an  additional  precaution  for  JASON  journal  tapes,  a 
brightly  colored  label,  identifying  current  JASON  journals, 
could  reduce  the  possibility  of  inadvertently  deleting 
important  JASON  data.  After  the  retention  period  on  JASON 
volumes  expires,  these  colored  indicators  should  be  removed 
before  releasing  JASON  tapes.  Additionally,  FEDSIM  recommends 
that  tapes  should  be  periodically  cleaned  and  recertified. 

The  use  rates  of  tapes  may  be  found  from  SCF  accounting 
data.  Each  time  a  tape  is  used  an  entry  is  created  in  the 
activity  accounting  record.  These  data  may  be  reduced  to 
record  the  frequency  of  use  and  the  u:,er  of  each  tape. 

Cleaning  and  recertification  of  tapes  minimizes  tape  errors. 

FEDSIM  recommends  that  NSF  management  establish  a 
computer  management  reporting  and  performance  evaluation 
section.  This  section  should  be  responsible  for  (1)  the 
collection  and  maintenance  of  accounting  and  software  monitor 
data  and  (2)  management  reporting  concerning  system  use  and 
availability.  Additional  functions  that  could  be  included 
within  t!iis  section  are  JASON  data  base  administration 
functions  and  approval  of  software  design  of  batch  and  on¬ 
line  applications.  Prior  approval  of  application  design 
will  ensure  that  these  applications  do  not  needlessly  impact 
system  turnaround,  respose  times,  and  the  availability  of 
system  resources. 

FEDSIM  noted  that  NSF  management  expressed  great  interest 
in  enhanced  information  with  which  to  manage  personnel  and 
computer  resources.  Much  of  this  information  currently 
exists  in  accounting  data  and  software  monitor  output. 

Personnel  in  the  management  reporting  section,  with  appropriate 
guidance  from  NSF  management,  should  use  these  data  to 
create  a  management  reporting  system  tailored  to  NSF  needs. 
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B.  WORKLOAD 


FEDSIM  recommends  that  NSF  personnel  establish  response 
and  turnaround  requirements  for  production  users  of  the 
H-6060  computer  system.  These  response  and  turnaround 
requirements  can  be  used  to  measure  the  performance  and 
capacity  of  the  system  and  to  select  certain  workloads  for 
execution  during  non-peak  processing  periods.  Findings 
indicated  that  much  batch  workload  ran  during  the  day  shift 
when  on-line  systems  were  active.  Identification  of  batch 
workloads  that  may  be  deferred  to  non-prime  processing 
periods  would  free  day  shift  processing  resources.  Findings 
have  shown  that  the  NSF  computer  system  has  excess  capacity 
during  evening  and  night  shift  processing  periods  to  process 
additional  workloads.  FEDSIM  does  not  recommend  that  NSF 
stop  processing  batch  workloads  during  the  day  shift. 
Findings  have  shown  that  peaks  and  valleys  exist  in  both  CPU 
and  memory  utilization.  Background  batch  activity  serves  to 
"fill  in"  the  valleys  in  machine  utilization,  resulting  in 
greater  throughput  with  little  or  no  degredation  in  response 
times.  Turnaround  and  response  requirements  serve  to 
measure  the  amount  and  type  of  batch  workload  that  could  be 
applied  during  the  day  shift  before  degradation  occurs.  NSF 
personnel  should  design  day  shift  system  scheduling  around 
the  primary  philosophy  of  running  priority  workloads.  Since 
most  workload  required  small  amounts  of  resources,  NSF 
personnel  should  only  consider  scheduling  based  on  resource 
utilization  for  activities  that  require  unusual  resource 
requirements.  Long-running,  low  priority  workloads  that 
require  little  operating  system  support  may  prove  to  be 
better  suited  to  run  with  on-line  systems  then  many  small, 
short  activities  that  require  more  operating  system  activity 
for  scheduling,  allocation,  and  termination. 

FEDSIM  recommends  that  the  Office  Management  System  be 
placed  under  centralized  control  and  be  modified  to  more 
closely  reflect  the  activity,  response,  and  turnaround 
requirements  of  system  users.  Four  main  points  were  iden¬ 
tified  in  interviews  with  Office  Management  System  users: 

(1)  users  were  required  to  group  Office  Management  System 
input  because  of  limitations  imposed  on  the  number  of 
proposals  that  could  be  processed  by  the  system,  (2)  some 
users  were  limited  in  Office  Management  System  applications 
because  the  number  of  reviewers  required  per  proposal  was 
more  than  that  allowed  by  the  system,  (3)  users  were  appre¬ 
hensive  about  using  the  Office  Management  System  for  a  large 
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number  of  proposals  since  no  restart  capability  was  provided, 
and  (4)  most  Office  Management  System  users  favored  the 
ability  to  quickly  enter  and  validate  proposal  and  reviewer 
numbers,  but  noted  that  the  letter  and  form  output  printed 
by  the  system  was  not  needed  by  users  until  the  next  day. 

FEDSIM  noted  from  workload  findings  that  the  Office 
Management  System  also  contributed  significantly  to  the 
number  of  characters  transmitted  by  the  TSS  system. 

FEDSIM  recommends  that  the  Office  Management  System  be 
redesigned  into  two  subsystems.  One  subsystem  would  operate 
on-line  under  TSS  in  much  the  same  way  as  the  present  system 
and  would  provide  on-line  entry  and  validation  of  proposal 
numbers  and  reviewers.  .The  other  subsystem  would  replace 
the  current  on-line  function  of  printing  large  volume  output 
at  low-speed  terminals  with  a  batch  oriented  subsystem. 

This  batch  subsystem  would  remove  much  Office  Management 
System  workload  from  TSS  and  direct  it  to  high-speed  printing 
devices  with  full  restart  capability. 

Figure  V-5  shows  a  suggested  implementation  of  the 
Office  Management  System.  Users  interface  with  a  TSS 
subsystem  to  enter  proposal  and  reviewer  information  and 
direct  the  type  of  forms  to  be  produced.  This  TSS  subsystem 
then  generates  batch  jobs  for  printing  each  type  of  form  and 
notifies  the  user  of  the  identification  of  each  job.  The 
batch  system  input  program  examines  these  jobs  (via  . MSCAN 
and  . MSCN1  to  identify  special  $IDENT  card  identifiers  or  a 
special  §  FORMS  card)  and  places  them  on  system  scheduler 
queues  corresponding  to  the  type  of  output  desired  (labels, 
special  forms,  etc.). 

During  non-prime  processing  periods,  the  computer 
operator  can  mount  special  printer  forms  and  then  run  jobs 
from  these  special  scheduler  queues.  Alternately,  since  the 
Job  Control  Language  for  these  jobs  is  pre-determined  with 
the  TSS  subsystem,  output  from  these  programs  may  be  assigned 
to  fictitious  terminal  identification  codes  corresponding  to 
the  type  of  output  desired.  The  operator  may  then  (1)  re¬ 
direct  output  for  these  station  codes  to  on-line  printers 
for  labels  and  forms  and  (2)  purge  reviewer  letter  output  to 
tape  for  printing  on  the  XEROX  high-speed  printer.  Under 
software  control,  this  printer  is  capable  of  supplying 
letterheads,  selectable  printing  fonts,  and  extremely  high- 
quality  output.  This  design  of  the  Office  Management  System 
places  responsibility  for  system  output  within  computer 
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operations  instead  of  with  the  computer  user.  Also,  this 
design  significantly  reduces  the  printing  load  on  the  TSS 
system.  FEDSIM  feels  that  implementation  of  this  recom¬ 
mendation  would  allow  significant  increases  in  Office 
Management  System  use,  as  requested  by  NSF  personnel,  with¬ 
out  serious  response  and  turnaround  degradations  to  on-line 
system  users. 

FEDSIM  recommends  that  batch  users  maintain  production 
programs  on  system  loadable  user  libraries  instead  of  on 
object  form.  Findings  have  shown  that  almost  all  program 
executions  are  accomplished  through  GELOAD,  the  system 
loader.  Loading  requires  extra  processing  resources  each 
time  programs  are  run  to  link-edit  each  program.  Additionally, 
new  operating  system  releases  may  cause  problems  with  program 
executions,  since  a  source  module  compiled  under  one  release 
(kept  in  object  form)  and  then  link-edited  with  a  system 
library  from  a  new  release  may  yield  unpredictable  results. 

Use  of  user  libraries  would  also  enhance  system  accounting 
since  the  activity  identification  would  not  be  GELOAD  but 
rather  the  actual  name  of  the  production  program.  Care  in 
assigning  unique  production  program  names  could  allow  the 
resource  use  and  run  frequency  of  individual  programs  to  be 
monitored. 

FEDSIM  recommends  that  NSF  personnel  investigate  the 
large  number  of  BASIC  programs  run  under  the  TSS  system.  If 
these  programs  are  used  repeatedly,  they  should  be  re-coded 
(e.g.,  into  FORTRAN),  and  be  made  into  loadable  TSS  sub¬ 
systems.  Currently,  FEDSIM  feels  that  much  CPU  and  I/O  time 
is  being  used  by  the  TSS  system  to  compile  BASIC  programs 
each  time  they  are  run.  As  a  minimum,  BASIC  users  should 
use  RUN=filename  to  save  the  object  module  generated  from 
BASIC  compilations.  Subsequent  executions  of  the  program 
will  require  only  loading. 

FEDSIM  recommends  that  NSF  personnel  investigate  the  use 
of  word  processing  equipment  in  lieu  of  using  the  TSS  Text 
Editor  for  the  creation  and  maintenance  of  large  document 
files.  Text  Editor  functions  such  as  "Find  String,"  which 
require  character-by-character  searches  of  large  data  files, 
impose  a  significant  load  on  TSS  processing  and  I/O  resources. 
When  large  data  files  are  edited  on  the  H-6060,  FEDSIM 
recommends  that  TSS  users  use  "Line”  mode  to  position  file 
pointers  close  to  the  page  or  paragraph  to  be  searched,  then 
use  "String"  mode  to  find  the  required  word  or  phrase. 
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FEDSIM  recommends  that  NSF  personnel  examine  the  possi¬ 
bility  of  removing  large-volume  data-entry  functions  from 
competing  with  other  JASON  users.  Although  NSF  has  imple¬ 
mented  a  form  of  this  recommendation  by  placing  certain  data 
in  staging  areas  within  the  JASON  database,  data  entry  users 
must  still  compete  for  memory,  CPU,  and  I/O  resources  within 
the  JASON  executive.  Through  user  interviews,  FEDSIM  found 
that  most  data  are  not  needed  for  processing  the  same  day 
they  are  entered  into  the  system.  FEDSIM  recommends  that 
data  entry  functions  be  implemented  under  TSS  (requiring 
little  or  no  change  to  user  interface)  and  be  stored  in  a 
non-JASON  file  for  later  rapid  update  of  the  JASON  database 
after  the  day  shift.  The  JASON  data  base  would  then  be 
available  for  processing  by  batch  programs  later  in  the 
evening  and  for  on-line  inquiries  the  following  day. 
Implementation  of  this  recommendation  would  provide  a 
quicker  and  more  consistent  response  time  for  data-entry 
functions. 

FEDSIM  recommends  that  the  JASON  system  be  modified  to 
provide  better  accountability  for  resources  used  and  system 
response  times.  FEDSIM  feels  that  the  current  information 
reported  by  the  JASON  system  concerning  user  activity  (user 
identification,  procedure  name,  terminal  identification, 
date,  start  time,  and  stop  time)  is  an  excellent  start  for 
tracking  JASON  activity  but  should  be  expanded  to  include 
the  accumulated  processor  time,  number  and  type  of  data  base 
calls,  number  of  terminal  characters  sent  or  received, 
amount  of  memory  used,  and,  if  possible,  keywords  used 
within  the  procedure.  The  information  can  be  used  to  identify 
procedures  that  use  large  amounts  of  system  resources  for 
possible  re-evaluation  and  optimization.  FEDSIM  also  recommends 
that  a  special  accounting  record  be  implemented  within  JASON 
to  track  system  response  time.  This  accounting  record  would 
be  written  each  time  a  response  request  was  received  by  the 
JASON  Executive  and  each  time  a  response  was  transmitted 
back  to  the  user.  Figure  V-6  presents  a  sample  accounting 
record  format.  Since  this  accountability  generates  extra 
overhead,  the  record  should  be  optional  so  that  it  may  be 
turned  on  or  off  as  needed  to  trace  JASON  response.  Data 
reduction  software  should  be  written  to  sort  records  based 
on  transaction  numbers  and  to  determine  response  time  on  a 
transaction-by-transaction  basis.  Additionally,  when  response 
degradations  do  occur,  this  record  provides  detailed  data 
about  what  transactions  were  in  process  at  the  time  of  the 
degradation  and  can  help  NSF  personnel  identify  functions 
that  adversely  affect  JASON  response. 
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C.  EFFECTS  OF  WORKLOAD  ON  SYSTEM  RESOURCES 

FEDSIM  recommends  that  NSF  personnel  periodically  add 
the  MCOUNT  patch  to  the  dispatcher  to  count  module  loads. 
Highly  used  re-entrant  modules  should  be  placed  in  the  Hard 
Core  Monitor,  thus  eliminating  large  amounts  of  disk  activity. 
FEDSIM  also  recommends  that  the  SSA  Cache  Memory  be  maintained 
on  the  H-6060  computer  system  to  further  reduce  the  need  for 
acquiring  system  modules  from  disk. 

FEDSIM  recommends  that  the  current  patches  concerning 
Priority  B  dispatching  be  maintained.  These  patches  give 
the  JASON  on-line  system  first  priority  with  a  Priority  B 
dispatch  interval  of  224  milliseconds  and  TSS  second  priority 
with  a  dispatch  interval  of  96  milliseconds.  FEDSIM  noted 
that,  at  current  user  activity  levels,  JASON  response  is 
more  a  function  of  the  interaction  between  JASON  procedures 
than  of  CPU  resource  availability.  System  loading  experiment 
#2  showed  that  overall  JASON  response  could  be  adversely 
affected  by  dispatching  parameters.  FEDSIM  identifies 
certain  changes  within  the  dispatcher  that  would  ensure 
priority  dispatching  to  JASON  when  needed.  The  following 
patch  to  the  GCOS  dispatcher  will  allow  Priority  B  programs 
to  accumulate  more  time  in  Priority  B  status  than  is  usually 
allowed  through  normal  system  parameter  settings: 

10132  OCTAL  000013735200  ALS  11  .MDISP 

This  patch  will  reduce  the  time  quantum  subtracted  from  the 
amount  of  Priority  B  dispatch  time  alloted  to  Priority  B 
programs.  NSF  personnel  may  want  to  alter  the  contents  of 
this  patch.  Each  bit  shifted  left  less  than  12  bits  divides 
the  current  quantum  used  by  one-half.  This  patch  will 
ensure  that  the  JASON  on-line  system  receives  adequate 
Priority  B  status  before  relinquishing  to  non-Priority  B 
programs. 

FEDSIM  identified  that  under  release  3/1,  TSS  subsystems 
are  dispatched  ahead  of  Priority  B  programs.  Depending  on 
the  savings  realized  by  recommendations  concerning  TSS 
subsystem  activity,  the  following  patch  to  the  TSS  executive 
can  increase  the  amount  of  CPU  resources  available  to  the 
JASON  system,  yet  not  seriously  degrade  TSS  response: 

2626  OCTAL  000001000000  1  PROC  SUBDISPATCH  .MTIMS 

This  patch  restricts  TSS  subsystem  execution  to  only  one 
processor  instead  of  allowing  TSS  subsystems  to  consume  the 
resources  of  both  processors  simultaneously.  With  some 


64 


ATTACHMENT  VI 


ATTACHMENT  VI 


minor  differences  imposed  by  system  release  3/1,  this  patch 
will  cause  the  GCOS  dispatcher  to  allocate  processing 
resources  to  programs  in  a  similar  manner  as  release  2/H. 

These  patches  were  developed  for  the  GCOS  3/1  operating 
system  in  use  at  NSP  during  the  measurement  study  and  as 
such  are  considered  to  be  correct.  NSF  personnel  should 
follow  good  patch  implementation  procedures  when  installing 
these  patches.  Ose  of  these  patches  on  subsequent  operating 
system  releases  should  be  verified  by  NSF  personnel. 

FEDSIM  recommends  that  the  system  trace  routine  be 
turned  off,  except  when  these  traces  are  necessary  to 
analyze  recurring  system  failures.  FEDSIM  has  found  through 
previous  Honeywell  studies  that  running  with  full  trace 
capability  subtracts  significant  processing  resources  from 
user  workloads.  (Each  trace  requires  an  average  of  200 
microseconds  to  process.)  System  traces  may  be  turned  on 
when  system  problems  occur,  although  examination  of  the 
first  failure  will  lack  necessary  system  traces.  Alternately, 
FEDSIM  has  identified  certain  traces  that  are  normally  used 
when  system  problems  occur  yet  result  in  minimum  impact  on 
CPU  resources.  These  trace  tapes  are  0,  1,  2,  4,  6,  7,  10, 

11,  13,  14,  15,  16,  20,  21,  22,  23,  32,  33,  37,  46,  50,  51, 

52,  61,  65,  70,  74,  and  77. 

FEDSIM  recommends  that  the  activity  during  test  versions 
of  JASON  be  monitored  to  see  if  the  ♦■ime  periods  required 
for  JASON  testing  can  be  consolidated.  Currently,  test 
versions  of  JASON  execute  twice  daily.  FEDSIM  observed  on 
numerous  occasions  that  very  little  activity  occured  during 
theBe  test  periods,  yet  the  test  versions  of  JASON  imposed  a 
significant  bottleneck  on  memory  resources,  sometimes  for 
periods  of  over  one  hour. 

Consolidation  of  test  times  would  make  more  productive 
use  of  test  versions  of  JASON  and  free  significant  system 
resources  at  other  times  for  normal  batch  workload. 
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